Eleventy Webmentions

Dusty Candland | Fri Dec 27 2019 | eleventy, 11ty, webmentions, bridgy, microformats

Owning my content and making it easy to post are both important to me and I've been slowly working to do that more. Moving my blog to 11ty was a big step in that direction. Now with webmentions and bridgy it's easier to share content. This post talks about what I've setup to connect my blog to the indieweb and some of the closed networks.

The parts

https://11ty.dev builds my site.
GitHub Actions auto deploy on commit.
https://forestry.io and VIM are used to edit the content.
Webmentions are used to let other know I've posted something related.
Microformats help others to share and understant my content.
Webmention.io allows me to get webmentions.
https://Brid.gy helps connect Twitter, Mastodon, and Flickr

https://sia.codes/posts/webmentions-eleventy-in-depth/

https://indieweb.org/Webmention-developer

GitHub Actions

This part actually came last, but I should have setup something like it sooner. It seems like actions are available to everyone now so I thought it'd be a good way to go. It was pretty easy to get setup.

I did run into one issue with getting the last commit date when posting. I fixed that for now by using an older version of the checkout action.

Here's my config.

name: Node CI

on:
  push:
  schedule:
    - cron: '10 14 * * *'

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v1
    - name: Use Node.js 12.x
      uses: actions/setup-node@v1
      with:
        node-version: 12.x
    - name: yarn install, build
      run: |
        yarn install
        yarn build
      env:
        CI: true
    - name: Deploy to server
      id: deploy
      uses: Pendect/action-rsyncer@v1.1.0
      env:
        DEPLOY_KEY: $
      with:
        flags: '-ave --delete'
        options: ''
        ssh_options: ''
        src: 'build/'
        dest: 'candland@0bacf2d9.wpserverhq.com:~/public_html/'
    - name: yarn webmention ping
      run: |
        yarn prod:ping
      env:
        CI: true

Quickly it runs once a day and on commits. The once a day is so that it will update any webmentions that happened in the last day.

It pulls the code, builds 11ty, rsyncs to my server, and lastly pings any sites that support webmentions that have been modified since the last build date.

Webmentions & Brid.gy

I started with Webmentions Eleventy In Depth, thanks Sia! It's great resource to get started and allow people to send you webmentions. I used the code there pretty much as is.

Since there are not a lot of people using just this, https://Brid.gy helps move data between the social sites and your site. It needs your site to have Microformats markup in order to automatically post content to the supported sites.

After adding that markup you need to setup Bridgy for publishing and then send them a webmentions when you have new content. Automating this part required the most work and is still getting the kinks worked out.

A few things I learned...

Modified dates don't work when using GIT, which actually makes sense. I used the GIT commit date.
The best way to know when the site was lasted built and deployed is to add a date to the home page and check that.
When you check a link for a webmention endpoint, you should check the actual link, not the just domain. I thought the domain might be a good optimization to limit requests, but that turned out to be a waste. Don't optimize until you need to :p.

After I got some of that worked out. I added in the webmentions links for Bridgy as hidden links on more recent posts.

Bridgy can handle favorites and reply-tos, but I haven't done anything with those yet.

Automating webmentions out

Step 1 was to get a last published date on the site so I know when content is added or changed.

In my site config file I have a field to set the buildTime.

module.exports = {
  title: 'candland.net',
  description: 'Notes about things',
  url: 'https://candland.net',
  domain: 'candland.net',
  buildTime: new Date(),
  ...
}

Then in the home page template I added this so I could look up that date later.

<time itemprop="lastPublished" datetime="{{ site.buildTime | isoDate }}">Published: {{ site.buildTime }}</time>

Step 2 was to get a list of links that needed to be pinged.

I used 11ty linters for that. The linter runs after the content is built, but is missing the meta data. The meta data would nice so I'd have the permalink, but I can rebuild the link from the outputPath, which I do have.

I built this as an 11ty plugin. It's pretty ugly still!

const osmosis = require('osmosis')
const fs = require('fs')
const {exec} = require("child_process")

module.exports = {
  configFunction: function(eleventyConfig, options = {}) {
    function writeToCache(file, data) {
      const dir = '_cache'
      const fileContent = JSON.stringify(data, null, 2)
      if (!fs.existsSync(dir)) {
        fs.mkdirSync(dir)
      }
      fs.writeFileSync(file, fileContent)
    }

    function readFromCache(file, defaultData = {}) {
      if (fs.existsSync(file)) {
        const cacheFile = fs.readFileSync(file)
        return JSON.parse(cacheFile)
      }
      return defaultData
    }

    function getPublishedTime() {
      return new Promise((resolve, reject) => {
        let lastPublished = new Date(new Date() - (7*24*60*60*1000))
        osmosis.get('https://candland.net').
          find("time[itemprop=lastPublished]@datetime").
          set("lastPublished").
          data((data) => {
            console.log("Last Published: " + data.lastPublished)
            lastPublished = Date.parse(data.lastPublished)
          })
          .error(() => resolve(lastPublished))
          .done(() => resolve(lastPublished))
      })
    }

    const domainRx = "^https?://candland.net" // config

    this.pastDate = null
    function lookupPubTime() {
      if (this.pastDate) return Promise.resolve(this.pastDate)
      return this.pastDate = getPublishedTime()
    }

    function lookupGitMtime(inputPath) {
      return new Promise((resolve, reject) => {
      const prog = `git log -n1 --format=%ct -- "${inputPath}"`
        exec(prog, (error, stdout, stderr) => {
          if (error) {
            console.log(`ERROR Reading GIT commit time ${inputPath}`)
            console.log(error)
            return resolve(new Date())
          }
          const mtime = new Date(parseInt(stdout.trim()) * 1000)
          resolve(mtime)
        })
      })
    }

    // TODO: exclude collection pages.
    async function wmLinter(content, inputPath, outputPath) {
      if (process.env.ELEVENTY_ENV !== 'production') return

      const links = new Set()
      const pastDate = await lookupPubTime()
      const mtime = await lookupGitMtime(inputPath)

      // Check for mtimeMs > pingMs
      if (pastDate > mtime) return

      console.log("Getting Links " + inputPath + " - " + mtime)

      return osmosis
        .parse(content)
        .set({'links': ['.h-entry a[href]@href']})
        .data((data) => {
          // console.log(data)
          const links = data.links

          const filtered = links.map((link) => {
            console.log("Found link: " + link)
            if ('string' === typeof(link) && link.startsWith('http') && !link.match(domainRx)) {
              const linkUrl = new URL(link)
              if (linkUrl.origin !== '' && linkUrl.pathname !== "/") {
                return linkUrl.href
              }
            }
          }).filter((link) => link !== undefined && link !== null)

          const unique = new Set(filtered)

          if (unique.size > 0) {
            const pings = readFromCache("_cache/pings.json", {})
            if (!pings[outputPath]) pings[outputPath] = {}
            pings[outputPath].mtime = mtime
            pings[outputPath].links = Array.from(unique)
            writeToCache("_cache/pings.json", pings)
          }
        })
        .error(console.log)
    }

    // Overwrite any existing cache
    writeToCache("_cache/pings.json", {})
    eleventyConfig.addLinter("webmentions-ping-dev", wmLinter.bind(this))

  }
}

For any content that's been modified after the last publish date I find all the links in the h-entry section of the page and write them to a JSON file with page as the key.

Step 3 is to finally send a webmention request to any link that supports it.

After the build another script runs that reads the JSON from step 2, looks up the webmention endpoint for each link and pings anything that has one. Also still not cleaned up :/.

const osmosis = require('osmosis')
osmosis.config('tries', 1)
osmosis.config('concurrency', 1);
osmosis.config('timeout', 10000);

const fs = require('fs')
const needle = require('needle');

class Ping {
  constructor(origin, removePath) {
    this.lookup = {}
    this.origin = origin
    this.removePath = removePath
    this.removePathLen = removePath.length
  }

  readFromCache(file, defaultData = {}) {
    if (fs.existsSync(file)) {
      const cacheFile = fs.readFileSync(file)
      return JSON.parse(cacheFile)
    }
    return defaultData
  }

  lookupEndpoint(url) {
    if (!this.lookup[url]) {
      this.lookup[url] = this.findEndpoint(url)
    }
    return this.lookup[url]
  }

  findEndpoint(url) {
    // console.log("Find endpoint for : " + url)
    return new Promise((resolve, reject) => {
      let endpoint = null
      osmosis.get(url)
        .find("link[rel=webmention]@href")
        .set("endpoint")
        .data((data) => {
          // console.log("Found endpoint: " + data.endpoint)
          endpoint = data.endpoint
        })
        .error(() => resolve(endpoint))
        .done(() => resolve(endpoint))
    })
  }

  pingEndpoint(endpoint, source, target) {
    needle('post', endpoint, {source: source, target: target}).then((rep) => {
      console.log(`Ping ${endpoint} with ${source} ${target}`)
    })
  }

  makeLink(outPath) {
    // Need to add two because the path starts with './'
    let trimmedPath = outPath.slice(this.removePathLen + 2)
    if (trimmedPath.endsWith("/index.html")) {
      trimmedPath = trimmedPath.replace("/index.html", "/")
    }

    console.log(`${this.origin}/${trimmedPath}`)
    return `${this.origin}/${trimmedPath}`
  }

  run() {
    const pings = this.readFromCache("_cache/pings.json", {})

    const all = Object.entries(pings).map((obj) => {
      const outPath = obj[0]
      const data = obj[1]

      const links = data.links.map((link) => {
        return this.lookupEndpoint(link).then((endpoint) => {
          if (endpoint) {
            const src = this.makeLink(outPath)
            this.pingEndpoint(endpoint, src, link)
          }
        })
      })

      return Promise.all(links)
    })

    Promise.all(all).then(() => {
      console.log("DONE")
    }).catch((e) => {
      console.log(e)
    })

  }
}

const prefix = process.env.NODE_ENV === "production" ? "build/" : "_site/"
new Ping("https://candland.net", prefix).run()

Misc Stuff and Links

I added a "new" format to my posts, called snips for small things like tweets and bookmarks. We'll see how well that actually works over time.

Webmentions are just HTTP Posts:

curl -i -d source=source -d target=target endpoint

Implementing Webmentions
IndieWebify.me Has nice validation tools.
Testing Webmentions

Other todos:

Webmention lookup is too limited. Only looks for a link tag with rel="webmention".
No support for deleting posts and sending mentions.
Need to exclude "index" page when finding links.

Webmentions

These are webmentions via the IndieWeb and webmention.io. Mention this post from your site: