Deepfakes have been discussed for a long time, and have been flagged as a major problem by many prominent people. I stumbled across it again in one of the recent episodes of the Waking Up podcast. I can't remember if it was the one with Marc Andreessen or the one with Eric Schmidt. Either way, even these highly technologically advanced people seem to miss an obvious mitigation to this problem; digital signatures.

By having prominent institutions, content creators and people (everyone?) provide a cryptographically signed digital signature along side the content they publish - anyone can verify the signature. They will of course have to decide for themselves if they trust institution, creator or person, but they can at least verify the content's origin.

If a piece of content shows up without, or with a completely new, digital signature, that would be a sign to be very skeptical of this content.

If a piece of content shows up from a prominent content creator with a valid digital signature, one that has been used multiple times in the past etc. we can be less skeptical of this content.

None of this is absolute. Private keys can be lost, content creators can be pressured, enticed or mislead, etc. So we still need to tread carefully. But it seems to me this would be a very good line of defense against deepfakes and also other "fake news".

And a lot of this stuff can be automated but digital tools and social media platforms!

As a fun anecdote, I once pitched this idea to Wikipedia's creator Jimmy Wales for his WikiTribune. He seemed to think it was a non-problem 😅 🤦🏻‍♂️

enjoy.

In the previous post here we introduced TezID and why we built it.

In this post we will outline how to use TezID.

We will use an example of a ICO contract that required participants to have 2 valid TezID proofs in order to qualify for registration. We will use SmartPy code in our examples.

Let's dive right into code!

## Types
#

TGetProofsRequestPayload = sp.TRecord(
    address=sp.TAddress, 
    callback_address=sp.TAddress, 
    callback_entrypoint=sp.TString
)
TGetProofsResponsePayload = sp.TRecord(
  address = sp.TAddress,
  proofs = sp.TMap(sp.TString, sp.TRecord(
    register_date = sp.TTimestamp, 
    verified = sp.TBool
  ))
)

## Contract
#

class ICO(sp.Contract):

  def __init__(self, tezid, requiredProofs):
    self.init(
      tezid = tezid,
      requiredProofs = requiredProofs,
      participants = {}
    )

  @sp.entry_point
  def signup(self):
    c = sp.contract(TGetProofsRequestPayload, self.data.tezid, entry_point="getProofs").open_some()
    sp.transfer(sp.record(address=sp.sender, callback_address=sp.self_address, callback_entrypoint="register"), sp.mutez(0), c)

  @sp.entry_point
  def register(self, ptr):
    sp.if sp.sender != self.data.tezid:
      sp.failwith('Only TezID can register')
    sp.set_type(ptr, TGetProofsResponsePayload)
    validProofs = sp.local("validProofs", [])
    sp.for requiredProof in self.data.requiredProofs:
      sp.if ptr.proofs.contains(requiredProof):
        sp.if ptr.proofs[requiredProof].verified:
          validProofs.value.push(requiredProof)
    sp.if sp.len(self.data.requiredProofs) == sp.len(validProofs.value):
      self.data.participants[ptr.address] = {}

Ok, so here we have some datatypes and a Tezos Smart Contract called ICO.

The datatypes TGetProofsRequestPayload and TGetProofsResponsePayload represent the structure of the data payload when calling and receiving a callback from TezID.

The ICO contract has the __init__ function to set some initial storage data. In addition is has two entrypoints; signup and register.

@__init__

Init is called when we create the contract. It sets some intial data for the contract;

* tezid ~ this is the address of the TezID contract
* requiredProofs ~ a list of required TezID proofs
* participants ~ the registered participants of the ICO

@signup

Signup can be called by anyone that attempts to sign up for the ICO. Signup will trigger a call to TezID's getProofs endpoint in order to retrieve the registered proofs for the calling address, and states that it would like it's own register endpoint to be called with this information (this is what we call a callback pattern, and it's how Tezos contracts can interact - atleast for now).

@register

Register is where TezID will call back with the registered proofs for some address. Note that it is set up to only allow TezID to call this endpoint. It received some proofs from TezID, compares to the required proofs set in requiredProofs, checks their validity and adds the address to the participants map if it all checks out.

And that is it 🎉

We now have created a contract that requires participants to have some proofs registered on TezID and we have more confidence that the registered participants are real people.

A SmartPy testcase

The test code for this ICO contract is quite long since it require all sorts of setup, but here it is if you are interested;

@sp.add_test(name = "Call TezID from other contract")
def test():
  admin = sp.test_account("admin")
  user = sp.test_account("User")
  user2 = sp.test_account("User2")
  user3 = sp.test_account("User3")
  cost = sp.tez(5)
  proof1 = sp.record(
    type = 'email'
  )
  proof2 = sp.record(
    type = 'phone'
  )
  proofVer1 = sp.record(
    tzaddr = user.address,
    type = 'email'
  )
  proofVer2 = sp.record(
    tzaddr = user.address,
    type = 'phone'
  )

  scenario = sp.test_scenario()
  c1 = TezID(admin.address, cost)
  scenario += c1
  c2 = ICO(c1.address, ["email","phone"])
  scenario += c2
  
  ## A user with the correct valid proofs can register as participant
  #
  scenario += c1.registerAddress().run(sender = user, amount = sp.tez(5))
  scenario += c1.registerProof(proof1).run(sender = user, amount = sp.tez(5))
  scenario += c1.registerProof(proof2).run(sender = user, amount = sp.tez(5))
  scenario += c1.verifyProof(proofVer1).run(sender = admin)
  scenario += c1.verifyProof(proofVer2).run(sender = admin)
  scenario += c2.signup().run(sender = user)
  scenario.verify(c2.data.participants.contains(user.address))
  
  ## A user without the correct valid proofs cannot register as participant
  #
  scenario += c1.registerAddress().run(sender = user2, amount = sp.tez(5))
  scenario += c1.registerProof(proof1).run(sender = user2, amount = sp.tez(5))
  scenario += c1.registerProof(proof2).run(sender = user2, amount = sp.tez(5))
  scenario += c1.verifyProof(proofVer1).run(sender = admin)
  scenario += c2.signup().run(sender = user2)
  scenario.verify(c2.data.participants.contains(user2.address) == False)
  
  ## A user not registered on TezID cannot register as participant
  #
  scenario += c2.signup().run(sender = user3)
  scenario.verify(c2.data.participants.contains(user3.address) == False)
  
  ## Only TezID can call register endpoiint
  #
  emailProof = sp.record(
      register_date = sp.timestamp(0),
      verified = True
  )
  phoneProof = sp.record(
      register_date = sp.timestamp(0),
      verified = True
  )
  proofs = {}
  proofs['email'] = emailProof
  proofs['phone'] = phoneProof
  pr = sp.record(address = user3.address, proofs = proofs)
  scenario += c2.register(pr).run(sender = user3, valid=False)

enjoy.

TezID is an identity oracle for Tezos 🤖✌️

It allows users to prove that they own certain digital property such as an email address, phone number, etc. And perhaps in the future a physical address and even government issued ID's.

How it works

At it's core TezID is a Smart Contract with registered addresses and verified proofs for each address. A user might register his/her address and different proofs, and use the oracle to verify these proofs.

The flow:

1. A user connects their wallet to the TezID dapp
2. The user registers their address with the TezID Smart Contract
3. The user registers one of the supported proofs with the TezID Smart Contract
4. The user requests a verification code from the oracle
5. The oracle verifies the uniqeness of the property (you cannot register the same email twice)
6. The oracle sends a verification code to the property
7. The user receives the verification code and enters it in the TezID dapp
8. The oracle marks the proof as verified in the Smart Contract

🙌

What is the point?

It can be used from other Smart Contracts to verify that a Tezos address is associated with a real human being.

I'm interested in collective decisionmaking mechnisms. Some of these require unique identities. One example is Quadratic Voting. An example usage of TezID would be to create a QV voting contract that will query the TezID contract if a certain address has registered a government issued ID proof within the last year, before allowing this address to vote.

But who knows what other use-cases might pop up 😬✨

What about security?

You might wonder if it's such a good idea to connect your Tezos address to your email or phone number etc. 🤔 This is not a good idea at all! 😳

Luckily this is not how TezID works. 😅 On the Smart Contract TezID only stores that you have registered a proof; it's type, the date this happened and a boolean to indicate if the proof has been verified by the oracle.

The TezID oracle only stores a hashed representation of your property. This means, even if the TezID oracle database was compromised, all an attacker would get was a list of hashes.

This is however not unproblematic. If an attacker get's ahold of the list of hashes, they could potentially attempt to hash known email addresses and compare to the list of hashes they have stolen.

Luckily this will not be so easy with the TezID hash table, since each hash is salted and generated using pbkdf2 (key stretching). We follow best practice principles outlined here. 🔒

We have done what we can to limit the attack surface of TezID.

Also check out our Tips section for how to avoid linking your TezID address with your other Tezos addresses.

Why is it costly to register proofs?

We have a added a cost for registering your address and proofs on TezID. At launch the cost will be around $10 for each, and we have the ability to adjust this over time.

We have introduced a cost here for several different reasons:

• To incetivice behaviour
• To fund running the oracle
• To fund further development of the oracle

To incentivice behaviour

It's super important to discourage people to register multiple addresses on TezID. The main idea is to have 1 Tezos address map to 1 human being.

For many of our proof types this is impossible to ensure. But having a high cost for registering proofs is a great way to discourage this behaviour. You can register 2-3 addresses, sure. But if you try to register 1000 it's going to cost you 😅💸.

Contracts using TezID should also require multiple proofs be registered and atleast one that is not easily duplicated.

To fund running the oracle

It's not free to run the TezID oracle. The cloud infrastructure has a cost. Sending out SMS'es has a cost. And the humans need to eat.

To fund further development of the oracle

We have lot's more ideas we want to realize with TezID, and any additionalt funds after running costs will go to further development.

Renewing proofs

The current owner of a property on TezID can renew it at any time. If a property has not been renewed for 365 days (depending on type of property) anyone in control of this property can use it to register a new proof.

It is a good idea for other contracts using TezID to require proofs less than a year old.

Calling from other contracts

You can see an example of how to use TezID from another contract here.

What about W3C Decentralized Identity?

We are aware of the work being done on W3C Decentralized Identity on Tezos spearheaded by Spruce Systems. We see TezID as supplemental to this work. On TezID you can verify your digital (and perhaps in the future physical) property. This makes TezID more of an issuer in a Decentralized Identity context. And who knows, perhaps one day someone will build a DID issuer layer on top of TezID 😬🙊

Looking forward to any and all feedback be it praise or critizism 🙌

✨🚀

Tips

Avoid linking your TezID to your other Tezos addresess

In order to register on TezID you need a Tezos address with some XTZ. You can just make a new one and transfer some Tez to it, but then your "main" Tezos address and all other addresses it has been in contact with (sendt to/from) will be linkable, and if just one of those addresses is connected to your personal identity, all the others will too 😳🙈 Such is the nature of public blockchains.

However, one "trick" you can do is to create an empty address. Then go on an exchange and purchase some XTZ. Transfer the XTZ from the exchange to your newly created address and register that address on TezID. That way there is no link between your other Tezos addresses and TezID. The exchange still has the information linking your identity and your TezID address, but there is no link to your other addresses on-chain.

enjoy.

Running Tezos on the Raspberry Pi 4.

I wanted to see if I could build and run Tezos on the RPi4. I'm currently running a bakery in the cloud; 4 nodes and a local signer. But cloudmachines has a running cost, atleast if you want somewhat beefy machine with a large SSD disk. RPi4 looks interesting since it now comes with a 4GB RAM model and a USB 3.0 interface for connecting a fast SSD.

UPDATED 25.11.2019

After discovering this bug I moved from the default raspbian image to Ubuntu 19.10 for Raspberry Pi running aarch64 instead of the 32 bit armv7. I'm also running the rootfs from the USB SSD instead of the MicroSD card which was a significant perf increase.

The hardware

• 4GB version Raspberry Pi4
• Crucial BX500 480GB 2,5" SSD
• ST Lab USB 3.0 to SATA 6G

The software

I initially used that standard raspbian distribution, and chose the Raspbian Buster Lite image since I need no desktop for this project.

However, after discovering a bug with tezos + 32 bit architecture, I moved to Ubuntu 19.10 for Raspberry Pi running aarch64 instead. Procedure is exactly the same; flash the image to MicroSD card and boot up.

At the time of writing the Ubuntu iso ships with a the 5.3.0-1007 kernel. This kernel has som issues with memory > 3GB, so if you have the 4GB version you need to mount the boot partition elsewhere and add the following to usercfg.txt:

total_mem=3072
dtparam=audio=on

There is a newer kernel available though; 5.3.0-1012 which fixes this issue. So if you update to it you can remove these lines and enjoy the full 4GB.

If you want to run the rootfs from the USB SSD (which I highly recommend since it's a real perf. boost), flash the exact same Ubuntu image on the SSD disk too. After that we just have to modify the root= of /boot/firmware/btcmd.txt (keep the rest):

root=/dev/sda2

Do that on both the MicroSD card's boot partition, and the SSD disks. Sometimes ubuntu mounts the SSD boot partition on /boot/firmare for some reason.

Only other software I installed was Docker. I ❤️ Docker. Cannot live without it. Get it!

curl -sSL https://get.docker.com | sh

So, I put together a declarative Dockerfile for Tezos that builds on arm. It's also available on Docker Hub.

It builds and works on the RPi4 😬🎉

Tezos

Next, let's make a few folders to keep the Tezos data:

mkdir -p /data/tezos/mainnet/client
mkdir -p /data/tezos/mainnet/node/data
mkdir -p /data/tezos/mainnet/snapshots

I want to run a full node, so let's get a snapshot so we don't have to sync the full chain. Find snapshot's here.

cd /data/tezos/mainnet/snapshots
wget <snapshot_url>

Next, let's load the snapshot:

docker run --rm \
-v /data/tezos/mainnet/client:/var/run/tezos/client \
-v /data/tezos/mainnet/node:/var/run/tezos/node \
-v /data/tezos/mainnet/snapshots:/snapshots \
--entrypoint bash \
-it asbjornenge/tezos-ubuntu-arm:latest

Notice I modified the entrypoint to run a container without running the entrypoint.sh script - it does currently not support loading a snapshot.

Next, let's make sure the permissions of our folders are correct:

conatiner> cat /etc/passwd | grep tezos

Note the uid and gid of the tezos user. Change the permissions (outside the container):

chown -R <uid>:<gid> /data/tezos/mainnet

Next, inside the container, load the snapshot:

container> tezos-node snapshot import /snapshots/mainnet.full --data-dir /var/run/tezos/node/data/

Before we can start the node we need to add the alphanet_version file (because of this bug):

echo 2018-06-30T16:07:32Z-betanet > /var/run/tezos/node/alphanet_version

Now we are ready to start the node!

docker run --rm \
-p 8732:8732 \
-v /data/tezos/mainnet/client:/var/run/tezos/client \
-v /data/tezos/mainnet/node:/var/run/tezos/node \
-v /data/tezos/mainnet/snapshots:/snapshots \
-it asbjornenge/tezos-ubuntu-arm:latest tezos-node

And thats it 😬🎉 Give it some time to start and you are running a full Tezos node on RPi4 🚀

Once it's started, you can verify the block level of your node:

curl -s localhost:8732/chains/main/blocks/head | jq .header.level

Performance

The CPU performance and MEM usage is well within bounds. The most important thing I wanted to perf test was the disk read/write speed.

I used a very basic read/write test to perf test the disk:

cd /data

sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.49814 s, 195 MB/s

dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.51642 s, 708 MB/s

As we can see we get a read speed og around 200 MB/s and write speed of around 700 MB/s. I was hoping for much better perf. than this, since both the SSD and adapter support SATA 3 and USB 3.1 "superspeed". But seems the limiting factor here might be that the RPi4 has a USB 3.0 port.

I was a bit encouraged by checking the read/write on my cloud nodes where both was only around 80 MB/s 😉

UPDATE 27.11.2019

Seems the above was also somewhat related to running the OS from the MicroSD card. After moving to Ubuntu and running OS from SSD I got the following results:

sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.7732 s, 225 MB/s

dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.819122 s, 1.3 GB/s

🎉🚀

Improvements

• Put OS on SSD ✅

That poor MicroSD card has disk read speed around 11 MB/s so I believe the reason the tezos-node RPC API sometimes feel a bit sluggish on the PI is caused by having the OS (and also Docker) running on that slow MicroSD card.

• UPS (powerbank battery) & 4G modem

I want to make the nodes as highly available as possible, so I want to add a batterybank and a 4G modem so it should be up no matter what (almost) 😉

• Signer with Ledger support

I haven't tested running a signer on the PI yet - it's on my TODO list.

Hope this was useful ✨

enjoy.

Sliq is a techelson testrunner for liquidity programs. It allows you to easily write and test smart contracts for the Tezos blockchain.

Sliq is a JavaScript cli application wrapping a docker image containing liquidity and techelson.

Another awesome thing is that liquidity supports Reason syntax, so now you can write both your contracts and your tests in Reason 💖

Install

Make sure you have node.js, npm and docker installed (and permissions to docker run without sudo).

npm install -g sliq

Use

sliq --contracts contracts/Demo.reliq --tests tests/

Sliq
  Contracts
    ./contracts/Demo.reliq
  Tests
    ./tests/Tests.reliq
Running tests...
===== ./tests/Tests.reliq =====
Running test `Sliq`

running test script...
   timestamp: 1970-01-01 00:00:00 +00:00

applying operation CREATE[uid:0] (@address[1], "tz1YLtLqD1fWHthSVHPD116oYvsd4PTAHUoc", None, true, true, 10000000utz) 
                       {
                           storage unit ;
                           parameter unit ;
                           code ...;
                       }
   timestamp: 1970-01-01 00:00:00 +00:00
   live contracts: none
=> live contracts: <anonymous> (0utz) address[2]
                   <anonymous> (10000000utz) address[1]

running test script...
   timestamp: 1970-01-01 00:00:00 +00:00

applying operation TRANSFER[uid:2] address[0]@Sliq -> address[2] 5000000utz "reason"
   timestamp: 1970-01-01 00:00:00 +00:00
   live contracts: <anonymous> (0utz) address[2]
                   <anonymous> (10000000utz) address[1]

running TRANSFER[uid:2] address[0]@Sliq -> address[2] 5000000utz "reason"
   timestamp: 1970-01-01 00:00:00 +00:00
=> live contracts: <anonymous> (5000000utz) address[2]
                   <anonymous> (10000000utz) address[1]

running test script...
   timestamp: 1970-01-01 00:00:00 +00:00

Done running test `Sliq`

NB! The first time you run sliq it pulls the required docker image from docker hub. It's about 182MB, so it takes a little while.

enjoy!

👀🔙👆

Chronos is a scheduled task runner built for docker cloud environments.

The common way to manage scheduled tasks is still to use cron. When managing a complex infrastructure this quickly becomes tedious. Keeping track of a bunch of cron jobs spread out across your servers - or even on a dedicated server introduces state, is hard to manage, monitor etc.

So I decided to do something about it... 🙈

Introducing Chronos 🥳🎉

Chronos is a scheduled tasks runner for docker cloud environments.

In Chronos you can add tasks to run at specific times defined in cron syntax. Each task can have multiple steps. Steps are executed in order and stdout and stderr are stored for each execution.

Steps are executed on the chronos-api service, it's a very basic alpine-linux container so it has very few tools for you to execute. It does however contain a docker cli - and the idea is for you to run scheduled tasks using containers.

State is stored in a postgresql database.

Hope you find it useful 🙌

enjoy.

Devops is currenly my bread and butter. For the different infra I manage I like to collect metrics using Prometheus.

When it came time to design and host a Tezos baking infra I wanted to collect baking metrics using Prometheus. As I could not find any solution already, I made one 😬🙌🌈🚀

https://github.com/asbjornenge/tezos-baking-exporter

I collect metrics from the RPC API of my own Tezos node(s). It's quite easy to work with and figure out 👍

It's still a WIP though, I know a couple of issues with how it collected metrics and presents them - but it work well enough to make informed decisions about the state of your node and baker 👍

React offers (imho) a paradigm shifting technology for client side web applications - or GUIs in general for that matter. If you haven't already; give it a try!

One of the things I really like about React is how it lends itself to TDD and testing in general. Small, focused components relying mostly on props (parameters) are easy to reason about and requires little mock.

The following are some of my experiences testing React components. Ready? Let's go!

React Test Utils

React ships with very good test utilities. Unfortunately the documentation is somewhat hidden away on their website. Here is a link.

var React          = require('react')
var ReactAddons    = require('react/addons') // You also need to require the addons
var ReactTestUtils = React.addons.TestUtils  // <- YEAH!

Use jsdom

For any kind of testing to be tolerable, TDD especially, efficient feedback loops are essential. Having to continously pass things off to a browser, or ever worse; multiple browser, is a pain. Luckily we have nodejs & jsdom. The React guys themselves use jsdom for testing.

I like to wrap up jsdom so that it is not required if a document already exists. That way the tests can run both in node and browsers.

UPDATE: There is a module for that (now) :-)

$ vi testdom.js
  module.exports = function(markup) {
      if (typeof document !== 'undefined') return
      var jsdom          = require("jsdom").jsdom
      global.document    = jsdom(markup || '')
      global.window      = document.createWindow()
      // ... add whatever browser globals your tests might need ...
  }

$ vi test/spec.js
  require('../testdom')('<html><body></body></html>')
  console.log(document)

Later we will see how we can hook up our tests to a CI tool for some sweet cross browser coverage.

Avoid JSX

It's up to you, but using jsx introduces an additional build step everywhere without adding much of a benefit. Using the React.DOM javascript API is really straight forward. It'll take you 2 minutes to figure out.

UPDATE: I've since changed my mind and are now using jsx. Partly because React introduce some breaking changes that would not have been a PITA if I had
been using jsx, and partly becuse it is a nice visual separation of logic and components.

Include a common render function

For each test you want a clean slate. Usually this means rendering the component again. It makes sense wrapping the render code into a function.

var _ = require('lodash') // or similar
var defaultProps = {}

function render(newProps, callback) {
    var props = _.merge(defaultProps, newProps)
    return React.renderComponent(Component(props), document.body, function() {
        if (typeof callback === 'function') setTimeout(callback)
    })
}

I find that keeping a set of defaultProps around makes sense. Callers of render can pass their required props (newProps) and have that merged with defaultProps before rendering. Overwriting the defaults if they want. Since we are testing components in isolation we can usually just mount to document.body. React.renderComponent takes a callback that is called when the component has finished rendering. I found that pushing my render's callback to the next tick of the eventloop (using setTimeout) resulted in a more stable test environment.

Clean up after each test

If you try to render a React component into a DOM which already has react identifiers, React will merge with whatever is already there. Especially when testing the same component over and over your need to clean up your DOM state.

How to do this depends on your test framework. Here is what I do in mocha (tdd interface):

describe('My Component', function() {

    afterEach(function(done) {
        React.unmountComponentAtNode(document.body) // Assuming mounted to document.body
        document.body.innerHTML = ""                // Just to be sure :-P
        setTimeout(done)
    })

    ...tests...
})

We use React.unmountComponentAtNode to unmount the component. Just to be safe we also reset body's innerHTML. I found once again that pushing the callback (done) to the next tick of the eventloop (using setTimeout) created a more stable test suite.

Query the DOM

You can query the DOM directly using the tool of your choice, or you can use the ReactTestUtils to query React components.

it('should render an input', function(done) {
    var _tree = render({}, function() {
        var __input = document.querySelectorAll('input')
        var _input  = ReactTestUtils.findRenderedDOMComponentWithTag(_tree, 'input')
        assert(...)
    })
})

As you might have noticed the findRenderedDOMComponentWithTag (and most other functions of ReactTestUtils) require a ReactComponent parent/tree to query. Luckily we designed our render function to return the top level component.

Simulate events

The ReactTestUtils also let's you simulate events. This is very useful!

it('should do something when I click mySpecialButton', function(done) {
    var _tree = render({}, function() {
        var _button = ReactTestUtils.findRenderedDOMComponentWithClass(_tree, 'mySpecialButton')
        ReactTestUtils.Simulate.click(_button)
        assert(...)
    })
})

For more about the capabilities of ReactTestUtils check out the docs.

Faking XMLHttpRequests

(Not really React specific, but I'll add a note about it anyway.)

Need to fake XMLHTTPRequests? There is a module for that!

var FakeXMLHTTPRequests = require('fakexmlhttprequest')

var requests   = []
XMLHttpRequest = function() { 
    var r =  new fakeXMLHttpRequest(arguments)
    requests.push(r)
    return r
}

describe('My component', function() {

    afterEach(function() {
        requests = [] // <- Reset request pool after each test
        ...
    })
    
    it('gonna get some data over the wire', function(done) {
        var onDataReceived = function(data) { assert(...); done() }
        render({ onDataReceived : onDataReceived }, function() {
            assert(requests.length > 0)
            requests[0].respond(200, { "Content-Type": "application/json" }, JSON.stringify({...}))
        })
    })

})

Running test

I use mocha solely for the nyancat reporter.

npm install -g mocha

I also find it useful to add my test command to package.json so that I can run my tests consistently with the same command across projects.

$ vi package.json
    ...
    "scripts": {
        "test": "mocha -R nyan -w --check-leaks",
    },
    ...

$ npm test

Testling

Running the tests in node is convenient and fast, but it is NOT THE SAME as running them in actual browser. So, we need to hook up some actual browser testing too. Testling is a great alternative and free for open source projects. They have great documentation and even a special little guide for using mocha. Plus, you'll get sweet badges:

There is one little trick I wanted to add though. Testling users browserify to create a browser compatible bundle of your javascripts. Unfortunately jsdom is not compatible with browserify, so we have to tell testling to ignore it.

In your package.json add a browser field and add tell browserify to ignore jsdom.

$ vi package.json

...
"browser" : {
    "jsdom" : false
},
...

Since we, in our jsdom wrapper above, only try to require jsdom if no document exists; the browser will never reach that code and we are good. The tests will use the browser's DOM.

Now go get some test coverage for your React components!

Credits

♥ to the React guys.
And the nyancat.
And coffee.
And tests.
And gifs.

enjoy!

Tired of having virtualbox and a thousand docker containers dragging your macbook air down in the mud? Put your docker host in the clouds!

In this post we will create a docker host running in the cloud and hook that up to our local development environment using a SSH based "VPN".

Her is a diagram.

    +---------------+               SSH                +----------------+
    |  development  | tun0         tunnel         tun0 |  docker host   |
    |               | <------------------------------->|                |
    |   (client)    | 10.0.0.1                10.0.0.2 | (cloud_server) |
    +-------+-------+     point to point connection    +-------+--------+
       eth0 |                                                  | eth0
192.168.0.2 |                                                  | 10.0.0.10
            |                    _  _                          | docker0
            |                   ( `   )_                       | 172.17.42.1
            -----------------  (    )    `)  -------------------
                              (_   (_ .  _) _)
                                  INTERNET

The Cloud Server

Pick a cloud, any cloud.

We'll stick with an Ubuntu 12.04 LTS on AWS EC2, but feel free to choose digitalocean or any other supplier and/or linux distribution. Create your VPS (follow provider instructions) and log in.

$ ssh <cloud_server_ip>

Prepare Docker

Follow the appropriate instructions to install docker. By default the docker daemon binds to a unix socket only. To be able to communicate with docker over the network, we need to bind to a network interface.

$ sudo vi /etc/init/docker.conf
	"$DOCKER" -d $DOCKER_OPTS -H unix:///var/run/docker.sock -H tcp://0.0.0.0:4243

WARNING! Binding to 0.0.0.0 will expose the docker host on all network interfaces on the server!! That might not be what you want. If you only want to talk to the docker host over the tunnel, you can bind to 10.0.0.2. Let's keep it exposed while we are testing, but remember to go back and fix that later.

Prepare SSH

To make this work, we need to permit root login over ssh. Now, that might seem like a security hazard. But, we are only going to allow login using keys (not passwords), and later on we are going to limit the access of root over ssh to only run a specific command.

$ sudo vi /etc/ssh/sshd_config
	PermitRootLogin yes
	PermitTunnel yes
	PasswordAuthentication no # <- optional, but recommended!
$ sudo service ssh restart

Friendly tip: When modifying sshd_config and restarting the ssh service; test your new configuration on a different session/terminal, keeping the one you already have open, just in case you messed up something.

We also need to remove any initial scripting under root's autorized_keys added to prevent login as root.

$ sudo vi /root/.ssh/authorized_keys

Remove anything resembling the following:

no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"ubuntu\" rather than the user \"root\".';echo;sleep 10"

...so that your file starts with rsa-ssh <long_key>.

Permit forwarding

$ sudo su -c "echo 1 > /proc/sys/net/ipv4/ip_forward"

The Client

If your on OSX (like me); install tuntap.

Grab the docker client!

$ curl https://get.docker.io/builds/Darwin/x86_64/docker-latest -o docker
$ chmod +x docker
$ sudo cp docker /usr/local/bin/

Thats it!

Connecting

Connection to your docker host is as simple as:

(client) $ sudo ssh -w 0:0 -i key.pem root@<cloud_server_ip>

Passing the -w flag to ssh will have ssh create a tunnel device on each end of the connection. The 0:0 indicates which tunnel device at each end. Since we have specified 0 at each end, both the device on the server and the client will be tun0. Now we need to route some traffic so we can talk to docker on the server.

Tunnels & Routes & Docker

Setup the tunnel devices:

(server) $ sudo ifconfig tun0 10.0.0.2 pointopoint 10.0.0.1
(client) $ sudo ifconfig tun0 10.0.0.1 10.0.0.2
(client) $ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=64 time=58.643 ms
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=58.410 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=58.586 ms
^ woohoo!

Setup routing:

(client) $ sudo route -n add -net 10.0.0.0 10.0.0.2
(client) $ sudo route -n add -net 172.0.0.0 10.0.0.2

Set the docker host:

(client) $ export DOCKER_HOST=tcp://172.17.42.1:4243

Et voilà:

(client) $ docker ps
CONTAINER ID        IMAGE                           COMMAND                CREATED             STATUS              PORTS

Addtional setup & security

Now, all this is quite a bit to remember, so I definately recommend scripting parts of the setup. One neat thing you can do that will both simplify the process and strengthen security is to only allow root to run a specific command over ssh, and have that command be opening the tunnel.

First, add the tun0 interface to /etc/network/interfaces

$ sudo vi /etc/network/interfaces
    iface tun0 inet static
	   address 10.0.0.2
	   pointopoint 10.0.0.1
	   netmask 255.255.255.0
	   # Forward traffic into server side network
	   iptables -t nat -A POSTROUTING --source 10.0.0.2 -j SNAT --to-source 10.0.0.10

Add a command to root's authorized_keys (first thing in the file) to bring up the interface on connection.

$ sudo vi /root/.ssh/authorized_keys
    tunnel="0",command="/sbin/ifdown tun0;/sbin/ifup tun0" ssh-rsa ....

Only allow commands for root over ssh.

$ sudo vi /etc/ssh/sshd_config
    PermitRootLogin forced-commands-only
$ sudo service ssh restart

Now, whenever you ssh in as root the server will try to bring up the tunnel interface.

DNS

If you want simple dns based service discovery for you containers over you new cloud bridge, apply the same tools and techniques discussed in my previous article Vagrant Skydocking to this cloud bridge. You will be pinging redis.staging.yourapp in no time.

Closing thoughts

I have been applying this technique for connecting to different VPCs I manage on amazon. Once scripted it's really nice being able to connect to a specific environment with one command and be hands on the docker host(s) and the containers in that environment.

Originally I had been hoping to also use a VPC for development purposes. Unfortunately the latency from my location to the datacenter gets annoying when trying to get efficient feedback loops. This is however not a fault of this approach, but the distance from me to the datacenter. Your milage may vary. Setting up a development host closer to home did the trick.

Time to dance!

Credits

♥ goes out to ssh.
And all the amazing unix hackers out there!
Including the authors of this and that.
Gifs from here.
Thank you internet of folks!

My favorite language at the moment is javascipt. It's fun & functional!

Since I'm also working quite a bit with docker, I've been frustrated with the size of nodejs docker images. A typical node container holds node, npm and all your dependencies. Add a few apt-get's and your quickly looking at > 500 MB.

I even started hacking some Go solely for the ability to compile to a single binary.

Until I found nexe...

Building with nexe

Nexe will compile your node app into a single executable binary. No joke! Have a look!

Since we are now compiling, we need to think about things like compile target. Containers run linux. My desktop runs Darwin. A binary compiled on/for Darwin won't be able run inside a container. So, I made a container for compiling apps with nexe.

docker run -v $(pwd):/app -w /app asbjornenge/nexe-docker -i index.js -o app

Weird bugs

Granted, nexe is a bit flakey atm. I found two main bugs that I had to work around:

A default package.json somehow messes up the executable.
Workaround: I added a build script that will move package.json to pkg.json, build, then move it back.

When passing arguments to a comiled binary, there must exist a first argument.
Workaround: Just pass a random first argument.

Container

When distributing, we can use the simplest container possible, and just add the binary.

FROM debian:jessie
ADD app /usr/bin/app
ENTRYPOINT ["app"]

Diff

I used this approach to build skylink, check out the difference!

      |   normal  |  nexe
 ---------------------------
 size |  640.3 MB | 133.6 MB

Credits

♥ to the nexe folks!
Gif from here.
Thanks!

I've been working quite a bit with docker lately. If you haven't yet checked it out, it's about time. Docker is already popping paradigms.

A bridge over vagrant water

Since I'm on OSX I'm running my docker host on Virtualbox via Vagrant.

Instead of having to forward ports and using lots of -p args when spawning containers, I wanted to bridge my host and the vm's docker interface, so that I could ping my containers from my OSX terminal.

Create a private_network in your Vagrantfile. I'm picking an ip on a different subnet than the docker0 interface to avoid any potential conflicts.

Vagrant::VERSION >= "1.1.0" and Vagrant.configure("2") do |config|
	config.vm.network "private_network", ip: "10.2.0.10", netmask: "255.255.0.0"
	config.vm.provider :virtualbox do |vb|
		vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
	end
end

The vb.customize is to allow forwarding packets for the bridge interface. The --nicpromisc2 translates to Promiscuous mode for nic2, where nic2 -> eth1. So --nocpromisc3 would change that setting for eth2, etc.

After reloading vagrant we need create a route on the host. Basically, any traffic trying to reach the docker subnet (172.17.0.0) should be routed to our new interface inside the vm (10.2.0.10).

# OSX
$> sudo route -n add -net 172.17.0.0 10.2.0.10
# Linux (untested)
$> sudo route -net 172.17.0.0 netmask 255.255.0.0 gw 10.2.0.10

You now have a bridge from your host to your docker network!!

$> IP=`docker inspect -format='{{.NetworkSettings.IPAddress}}' skydns`
$> ping $IP
PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data.
64 bytes from 172.17.0.3: icmp_req=1 ttl=64 time=0.232 ms
64 bytes from 172.17.0.3: icmp_req=2 ttl=64 time=0.103 ms
^C
--- 172.17.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1009ms
rtt min/avg/max/mdev = 0.103/0.167/0.232/0.065 ms

Skydock

Docker is all about distributed systems; packing single components inside containers and have them talk to eachother. One of the painpoints when shattering your monolith is linking all those loose components together.

(Docker provides a -link parameter for linking containers. But this quickly falls short in complex scenarios.)

I was just about to dig into service discrovery solutions like etcd or similar, when Michael Crosby posted his skydock (video). It's brilliant! It let's you discover your services via DNS. I won't go into setting up skydock, just check out the awesome tutorial by Michael.

So, with skydock my containers can discover eachother via DNS names like myservice.env.domain.com. Awesome! But, with my network bridge set up, so can my host!! No? That would be really nice for development...

$> curl elasticsearch.dev.domain.com:9200
curl: (6) Could not resolve host: elasticsearch.dev.domain.com

﴾͡๏̯͡๏﴿ ... Ah, we need to hook up skydns as a nameserver. This is where I stray a little from Michael's skydock tutorial. I had some issues binding to the docker0 interface (docker v0.7.6), so instead I'm using the skydns container as the nameserver directly (PS! this requires passing a -dns <skydns_ip> arg to each new container). Either way, we have to edit resolv.conf.

$> sudo vi /etc/resolv.conf
   # nameserver 172.17.42.1 <- skydock tutorial
   nameserver 172.17.0.3 # <- skydns container ip
$> dig elasticsearch.dev.domain.com
;; ANSWER SECTION:
elasticsearch.dev.domain.com.	20	IN	A	172.17.0.7

✌(-‿-)✌ ... Hoplah! Now, hopefully that will be it for you and you're all set to curl containers from the comforts of your host terminal! I however, had one more issue to solve...

$> curl elasticsearch.dev.domain.com:9200
curl: (6) Could not resolve host: elasticsearch.dev.domain.com # w00000000t???

OSX weirdness

Apparently OSX is rather weird in how it handles DNS. dig, host, etc. can resolve the host just fine, but other tools like curl and even ping does not obey resolv.conf. I eventually stumbled across the issue and found this script that apparently solves it for most people. It didn't help. Eventually I added the DNS server via OSX network preferences, and that did the trick.

$> curl elasticsearch.dev.domain.com:9200
{
	"ok" : true,
	"status" : 200,
	"name" : "Damian, Margo",
	"version" : {
		"number" : "1.0.0.Beta2",
		"build_hash" : "296cfbe390dc51bb00c00ba48ad0c8a9efabcfe9",
		"build_timestamp" : "2013-12-02T15:46:27Z",
		"build_snapshot" : false,
		"lucene_version" : "4.6"
	},
	"tagline" : "You Know, for Search"
}

I'm now a ᕙ༼ຈل͜ຈ༽ᕗ curl’er of containers!!

Credits

Docker, Skydock and Skydns all deserve a big fat ♥.
I followed this guide by Lukas Pustina to set up my vagrant networking.
Gifs from here and faces from there.
Thanks!

My first real fight with fonts.

Disclaimer

Before I even start this I should probably state that this adventure leads into unfamiliar terrain, and over half my findings are probably half-witted nonsense. There. I should probably start all my blogposts like that.

Introduction

Fonts are important. Most of what we see on our screens is text in some form, or typeface, to dive right into the syntax.

Starting out on my current font adventure I was quite shocked by how little I knew about the font world. I had been developing websites and apps full of text for years, but hardly knew what a baseline was. To some extent that is a good thing, it had just worked. On the other hand, it is a level of control over my design I had been completely ingorant about.

The design bullet

The current problem I was facing seemed simple enough; allow a line of text to include a bullet. Easy as pie.

<span style="display:list-item">Some Text</span>

Turned out it wasn't quite so easy. These bullets were part of our clients design manual, but they were not the same as the bullet glyph of the font. Modifying the font was also out of the question because of licensing.

But, there was a clear definition; the bullet was a square of height and width x relative the font-size, vertically aligned (centered) with the fonts [x-height](http://en.wikipedia.org/wiki/Baseline_(typography)).

Calculating the x-height of a target element is easy enough using css's ex unit.

$('<div style="width:1ex"></div>').appendTo(target)[0].offsetWidth

But the x-height itself was of little use. To vertically align my bullet with the x-height, I needed to know the margin, bottom or top, of the baseline or median; I needed more metrics.

Alright, easy enough. Let's see… "javascript font metrics". Uhm…

The bad news

There is no built-in, easy, standard way of extracting the metrics of a font.

The good news

It's possible to calculate! AND, there is a great library that will do most of the heavy lifting for you! We'll get to that.

Calculation

To calculate a fonts vertical metrics, there are two approaches as far as I can tell.

1. Measuring dom elements

The first approach is to use a bunch of dom elements with specific font-related metrics (1em, 1ex, etc.) and measure these in px (offsetWidth) at different levels and at different font-size's.

The approach seems to work quite well for the calculation part. Sturdy across browsers and fonts. For the actual positioning there were other icebergs floating around.

NB! The solution is a possible performance drain if used unwisely - measuring offsetWidth might cause unwanted reflow (repaint of your dom elements).

2. Canvas

The second approach is using the canvas element. The 2d context of a canvas has font, fillText and measureText functions. Unfortunately measureText only deals with the width metric, but that seems to be about to change (!!). For now though, the approach is to dump and analyze the raw pixel data and figure out how many pixels are used vertically to draw different letters of the font.

This approach also works perfectly for the calculation part, and thanks to the awesome fontmetrics.js it's easy.

But again, for the actual positioning, I was soon stuck in a pitch black room (next to a tiny, grey, startling little cat with diarrhea. Sitting on a matressless, iron-sprung bed with its huge eyes mewing at me. Meow. Smoking as well, probably. And then some terrible guy the colour of an aubergine round the corner holding a mug of beef tea and wearing a string vest going “meew. Fuckn brrr aaah” ~ Dylan Moran).

@font-face

The days of web typography is upon us. We are no longer limited to a handful of built-in fonts. Using technologies like @font-face we can embed "any" font on our page and have it render "beautifully" on the client's browser.

There are however quite a few pitfalls & legibility issues.

Rendering

The one that hit me hard in the face is the fact that different browsers, and even the same browsers on different operating systems, deal very differently with how they render fonts. Even different versions of the same operating system will sometimes render fonts very differently.

At typical body-text sizes, the computer has to draw each letter using only 15 or so pixels in each direction. It’s not possible to draw each letter exactly as the typographer intended, and keep all the lines crisp and smooth, with that few pixels. Windows, OSX, and Linux all resolve this dilemma differently: to oversimplify a bit, OSX tries harder to preserve the font shapes, Windows tries harder to make the lines sharp, and Linux tries to do both at once and winds up achieving neither.
~ Zachary Weinberg

Sometimes the font won't even render inside it's bounding box! (!!!!) For my current problem, that makes any font metric calculation futile. Turns out, this library I've been mumbling about had a solution for even this.

Timing

Another issue with embedded fonts is knowing when the font is loaded. If you try to measure prematurely you will end up measuring the fallback font, and thats no good.

The only viable solution I have come across is using a "dummy" fallback font that will encode a character as a zero-width unit. Putting that in a paragraph and polling for a real width. It's not a great solution but it works.

Font.js

Fortunately someone has already thread this path for us.
Font.js adds a Font object to your javascript toolbelt. It's designed to behave similar to the Image object.

var font = new Font();
font.onload  = function() {}
font.onerror = function() {}
font.src = "http://your.domain.com/fonts/font.otf"

It handles timing issue using the detailed solution above, and will call your onload function when the font is available. It gives you metrics.

font.metrics -> {}
font.measureText(string, size) -> {}

They even handle the rendering issue (to some extent).

Font.js actually draws text offscreen, does a scanline pass to find out what the "real" ascent and descent is, and then sets height to ascent + 1 + descent ("1" for the baseline itself). This generally works quite well, but will lead to incorrect heights for fonts that don't implement the Latin blocks =)
~ Michiel Kamermans

One important thing to note is that the fonts are loaded using XMLHttpRequest's. This is important since it is the only way to get the font data so it can be inspected and manipulated. But it does mean you have to deal with hosting your own fonts or setting up CORS to avoid Access-Control-Allow-Origin issues.

Font.js is a great library for solving most of the current headaches related to fonts.

Grab it from the github repo or via bower.

bower install Font.js

Resources

http://pomax.nihongoresources.com/pages/Font.js/
http://www.brunildo.org/test/xheight.pl
http://www.icavia.com/2010/09/solving-font-face-alignment-issues/
http://mudcu.be/journal/2011/01/html5-typographic-metrics/
http://www.owlfolio.org/htmletc/legibility-of-embedded-web-fonts/
http://en.wikipedia.org/wiki/Baseline_(typography)
http://stackoverflow.com/questions/1134586/how-can-you-find-the-height-of-text-on-an-html-canvas

Todo workflow for inbox zeroists.

I'm an inbox zeroist; my inbox is my todo list.

For us (well, for me at least) todo applications quikly get neglected. I love their shiny UI's and impressive and thought out UX, but the fact remains that the tasks I so optimistically punch in never get done. I've tried numerous approaches. My read-later services are filled to the brim by awesomeness that will never get parsed by anyone but @marcoarment's robots.

I always return to my inbox, so whatever gets in there gets action.

The following is an attempt to simplify adding "tasks" to my inbox.

Postfix

Get your local postfix relaying to a proper smtp server. I followed this guide for gmail. Be sure to also add the following to /etc/postfix/main.cf.

smtp_sasl_security_options = noanonymous

PS! See the update section at the bottom of this article.

$ mail

Now you can send emails from your shell.

df -h | mail -s "Disk usage" you@domain.io

Hotkey

There are multiple ways to have a hotkey execute a script. I choose Alfred because I like Alfred and because it has support for passing any selected text as an argument to the script.

Add your extension. It might be a good idea to click Advanced and configure escaping. Mail seems to handle all these chars nicely, so I just unchecked it all.

Add a hotkey for that extension and check "Selected text in OS X".

An thats it, you can now select any text in osx and stack it on top of your inbox by pressing your specified hotkey.

Update

For Yosemite I had to add the following to /etc/postfix/main.cf

smtp_sasl_mechanism_filter = plain

Found the solution here. Thanks!

Your probably talking JSON with a RESTful api, right?
If you care about creating a great experience, you need to take error handling seriously. Handling timeouts and http error codes is pretty straight forward, but handling corrupt data can be tricky. It often leaves an ugly footprint in your code. Lot's of if's and hasOwnProperty's. Instead, using json-schema, you can validate your JSON data first and be sure it is as expected.

JSON-Schema

A JSON Media Type for Describing the Structure and Meaning of JSON Documents

Example; If you have some JSON Data:

{
	"title" : "Kapsokisio"
}

You can define a corresponding JSON Schema:

{
	"type" : "object",
	"required" : ["title"],
	"properties" : {
		"title" : { "type" : "string" } 
	}
}

You can validate your data using that schema. If it is valid, you can be sure this data is an object with a title property of type string.

Specification

The latest IETF draft is currently v3, but they have a v4 being prepared for submission in early 2013. This post will focus on v4.

UPDATE
The new drafts are up!

Core:
http://tools.ietf.org/html/draft-zyp-json-schema-04
Validation:
http://tools.ietf.org/html/draft-fge-json-schema-validation-00

Software

There is a variety of implementations available. Since I choose to focus on v4 and since I'm a webnerd, I'll be using the tv4 validator for the examples.

Usage

NB! This article is in no way a usage reference!!
It's more a collection of the things I stubled across trying to figure out how this JSON-Schema thing works. Some important bits, and some of the things I found really useful. See the further reading section for more possibilites and options.

type

Using "type" you can specify the datatype required for the current object. The value can be a string or an array. Available values are; object, array, string, boolean, integer, number, null. The following requires the data to be either an object or a string.

{
	"type" : ["object","string"]
}

tv4.validate({}, schema) // true
tv4.validate([], schema) // false

enum

Using "enum" you can define an array with elements of any type. Data must be equal to one of the elements to validate.

{
	"enum" : [[1,true,0], {}, 28, "Burbon"]
}

tv4.validate([1,true,0], schema) // true
tv4.validate(34, schema) // false

required

Using "required" you can define an array of required properties. It's value is an array of strings.

{
	"required" : ["title","origin"]
}

tv4.validate({"title" : "", "origin" : ""}, schema) // true
tv4.validate({"title" : ""}, schema) // false

properties

Using "properties" you can further specify an objects properties. It is an object where each value is a separate schema.

{
	"properties" : {
		"title"   : { "type" : "string" },
		"weight"  : { "type" : "number" }
	}
}

tv4.validate({"title" : "", "weight" : 2}, schema) // true
tv4.validate({"title" : "", "weight" : "2"}, schema) // false

items

Using "items" you can specify the requirements for the items in an array. It can be a single schema or an array of schemas. The following requires the elements in this array to be a string or an object.

{
	items : [
		{ "type" : "string" },
		{ "type" : "object" }
	]
}

tv4.validate(["",{}], schema) // true
tv4.validate(["",true], schema) // false

pattern

Using "pattern" you can validate using regular expressions. Powerful stuff!

{
	"properties" : {
		"url" : { "type" : "string", "pattern" : /(http|ftp|https):\/\/[\w-]+(\.[\w-]+)+([\w.,@?^=%&:\/~+#-]*[\w@?^=%&\/~+#-])?/ }
	}
}

tv4.validate({"url" : "http://google.com"}, schema) // true
tv4.validate({"url" : "htt:/googleco.m"}, schema) // false

$ref

Using "$ref" you can reference other schemas. You can use a URI or an # for internal referencing. Using definitions as a location for your internal referenced schemas is not a rule but a common practice.

{
	"items" : { 
		"$ref" : "#/definitions/bean"
	},
	"definitions" : {
		"bean" : {
			"type" : "object",
			"required" : ["origin"],
			"properties" : {
				"origin" : { "enum" : ["kenya","rawanda"] }
			}
		}
	}
}

tv4.validate([{"origin" : "kenya"}], schema) // true
tv4.validate([{"origin" : "brazil"}], schema) // false
tv4.validate(["kenya","rawanda"], schema) // false

allOf

Using "allOf" you can define an array of schemas where your data elements must validate against all of them.

{
	"allOf" : [
		{ "type" : "integer" },
		{ "minimum" : 6 }
	]
}

tv4.validate(6, schema) // true
tv4.validate(5, schema) // false

oneOf

Using "oneOf" you can define an array of schemas where your data elements must validate against one (and only one) of them.

{
	"oneOf" : [
		{ "type"    : "integer" },
		{ "minimum" : 6 }
	]
}

tv4.validate(5, schema) // true
tv4.validate(6, schema) // false

anyOf

Using "anyOf" you can define an array of schemas where your data elements can validate against any (at least one) of them.

{
	"anyOf" : [
		{ "type"    : "integer"  },
		{ "minimum" : 6 }
	]
}

tv4.validate(5, schema) // true
tv4.validate(6, schema) // true

not

Using "not" you can define a schema your data elements should to not validate against.

{
	"not" : { "type" : "string" }
}

tv4.validate(1, schema) // true
tv4.validate("test", schema) // false

Error handling

(tv4 specific)

I just thought I'd quickly mention how tv4 handles a failure:

tv4.validate([],{"type" : "object"})
var err = tv4.error
while(err != null) {
	console.log(err.message, err.schemaPath, err.dataPath)
	err = err.subErrors
}

Further reading

I would really recommend reading through the tests for tv4, they provide excellent usage examples for the different possibilites. On the JSON-Schema website you will find the documentation and some great examples.

Pros

One of the biggest benefits of using JSON-Schema validation is that it will allow you a cleaner codebase. You can trust your data. That in turn improves readability and maintainability which leads to better and more robust applications. In the end; a better user experience.

Cons

It can be quite tedious building a good schema describing your data. And of course, if you change your data structures, you need to update your schema (in addition to your code). But considering how this approach will simplify your codebase, I would definately say it's well worth it.

Real world example

Data

{
	"title"   : "Kapsokisio",
	"origin"  : "Kenya",
	"variety" : ["SL28","SL34","Burbon"],
	"process" : "Washed",
	"roast" : {
		"level" : 4,
		"date"  : "08.02.2012"
	},
	"bag" : {
		"weight" : 354,
		"date"   : "08.02.2012"
	},
	"brew_tip" : {
		"method" : "pourover",
		"grind"  : "medium",
		"vessle" : "chemex"
	}
}

Schema

{
	"type" : "object",
	"required" : ["title","origin","variety","process","roast","bag"],
	"properties" : {
		"title"    : { "type" : "string"  },
		"origin"   : { "type" : "string"  },
		"variety"  : { "type" : "array"   },
		"process"  : { "type" : "string" },
		"bag"      : { "$ref" : "#/definitions/bag" },
		"roast"    : { "$ref" : "#/definitions/roast" },
		"brew_tip" : { "$ref" : "#/definitions/brew_tip" }
	},
	"definitions" : {
		"roast" : {
			"type" : "object",
			"required" : ["level", "date"],
			"properties" : {
				"level" : { "type" : "integer" },
				"date"  : { 
					"type" : "string", 
					"pattern" : /^\d{2}([./-])\d{2}\1\d{4}$/
				}
			}
		},
		"bag" : {
			"type" : "object",
			"required" : ["weight", "date"],
			"properties" : {
				"weight" : { "type" : "number" },
				"date"   : { 
					"type" : "string", 
					"pattern" : /^\d{2}([./-])\d{2}\1\d{4}$/
				}
			}
		},
		"brew_tip" : {
			"type" : "object",
			"required" : ["method","grind","vessle"],
			"properties" : {
				"method" : { "type" : "string" },
				"grind"  : { "type" : "string" },
				"vessel" : { "type" : "string" }
			}
		}
	}
}