picture of Josh Dzielak in 2016

Hi! I'm Josh Dzielak, writing here about technology and culture. Find me also on Twitter, Medium, Speakerdeck and Github.

weeve - HTML5 Twitter uber-streaming powered by Firebase, Keen IO, and Singly

weeve logo

Update: weeve has been retired, but the Keen and Firebase APIs allow you to build more apps just like it :)

Ever wish you could trade Twitter timelines with someone for a day? Or just a few minutes?

weeve goes one step further, allowing groups of Twitter users to combine their timelines to create an uber-timeline. Try it out - join the weeve at http://weeve.dzello.com/.

I created weeve in 2 days over the winter holiday as a fun way to get hands-on with some new tech. My goals for weeve were:

  • to play with some APIs I'd been hearing a lot about
  • to build a server-less app using a Backend-as-a-Service
  • to open-source the project for others to benefit from
  • to build something useful that could help people discover new content on Twitter
  • to distract me lest I consume hordes of delicious holiday foods

All but one of those goals was accomplished. :)

In this blog post I'll talk about how weeve works and APIs that made it possible.

The API's

These APIs, and the companies behind them, are doing great things. It's no coincidence I chose them to hack on. Here they are, including a brief summary of what they do in (mostly) my own words.

Firebase - https://firebase.com/

Firebase provides data storage that's directly and securely accessible from the browser. With Firebase, you lay out your data hierarchically, then bind clients to changes that happen at individual nodes. The unification of messaging locations and data is intuitive and powerful. And the JavaScript SDK is a pleasure to work with. The Security API has a learning curve, but it's a unique take on a important problem for server-less development - definitely worth spending the time to pick up.

Example code, taken from writing data in the Firebase docs:

// First we get a reference to the location of the user’s name data:
var nameRef =
  new Firebase(
    'https://SampleChat.firebaseIO-demo.com/users/fred/name');

// And then we write data to his first and last name locations:
nameRef.child('first').set('Fred');
nameRef.child('last').set('Swanson');
Keen IO - https://keen.io/

Keen IO does analytics by API. It's a developer-friendly solution that solves both problems - collecting event data and looking at event data. Recently, Keen IO released a JavaScript Visualization API that lets you draw charts & graphs with just a few lines of code.

Example code, taken from weeve:

// Draw a pie chart in just 2 function calls!
new Keen.Metric("tweets", {
  analysisType: "count", groupBy: "source_screen_name"
}).draw($("#top-weevers")[0], {
  showLegend: true, title: "Top weevers today"
})
Singly - https://singly.com/

Singly is the intelligent, unifying proxy for social network APIs you asked Santa for this year. Singly already has support for over 35 services. One thing I love about Singly is that their APIs are open source.

And if that wasn't enough, Singly is also behind The Locker Project, which has the bad-ass mission of making it possible for you to keep track of the mountains of personal data you spread around the nets. It's like insurance for the next time this happens: "All Online Data Lost After Internet Crash".

Example code using the profile Singly endpoint:

// Retrieve a user's unified profile
$.getJSON("https://api.singly.com/profile"),
  { accessToken: "HOHOHO" },
  function(profile) {
    alert("Would "  + profile.name + " like some fruitcake?")
  }
)

Special mentions

The following services and libraries are also used by weeve.

Twitter - https://twitter.com/

When people say 'the twitters' they really just mean Twitter.

jQuery, Bootstrap, Backbone, twitter-text, socket.io, CDNJS & more

Get the full list on the Github README.

How weeve works

weeve has no server. weeve is just an HTML page, a JS file, and a CSS file. You can deploy weeve under to any static web server like Apache and nginx, or to Github Pages.

Despite being flat, weeve has a host of features that traditionally require a server:

  • OAuth Authentication

    Singly proxies the OAuth conversation with Twitter. Then, Singly creates a Firebase authentication token and hands it back to weeve as a location fragment. The weeve Backbone router uses the token to establish a faux 'session' backed by localStorage, so the authentication persists across page refreshes. Read more about this flow here - Firebase Authentication Setup and here.

  • Central data storage

    Firebase is used to store data in a way that's accessible to all clients, yet still secure. User A cannot tamper with User B's data, etc. See the weeve security rules JSON definition to see what's defined.

  • Shared messaging layer

    Clients get notifications about what's going on by binding to Firebase data references. Everything happens in real-time and neatly stays in sync. weeve uses Firebase 'on' bindings to reflect the presence of users and to add tweets to the uber-timeline.

  • Analytics

    Application events are logged to Keen IO via their Javascript SDK. weeve uses the combined Keen IO SDK to get event logging and metrics/visualization all in one package. The event logging piece is also available separately. The SDK lets you define metrics, including timeframes, intervals, and group-by, and then makes it easy to represent the metric as a number, pie chart, or line graph.

    weeve uses Keen IO to display who's contributing the most tweets (pie chart), how many total weevers there are (number), and recent tweet volume as a series (line graph).

  • Twitter's Streaming API

    weeve uses the user stream endpoint of the Twitter streaming API. Tweets appear instantaneously; as in the instant someone pushes 'send'. It's quite thrilling.

    (I did have to write a 40-line node.js twitter streaming proxy to make the streaming API accessible from the browser. In the future, I hope Singly's experimental push support can replace the proxy!)

Get the code

weeve is open source. You can read the annotated source code and learn how to host your own weeve(s) on Github at dzello/weeve.

If you're looking to build something with one or more of these APIs, definitely check out the repo - there's a good chance you might find a relevant example of something you'd like to do.

weeve.js is the main (and only!) JavaScript file. I've added lots of comments and taken extra care to lay the code out in a way that's easy to follow.

Conclusion

I'm very happy with the way things turned out. I started with quite a few unknows and estimated at least a few unsightly hacks would be required to get it all working.

The streaming proxy was the only 'hack' required. I think that says a lot about the feasability of these API's for pure-client-side development and the readiness of backend-as-a-service providers for real-world HTML5 apps.

weeve is only a few hundred lines of JavaScript. And of that maybe 100 lines aren't 'boilerplate'. Needing just 100 lines of 'reasoned' code to get this feature set is mind-blowing.

And what's positively mind-altering is that there's no server to scale, deploys are sub-second (cp baby), and the heavy lifting is distributed across the browsers of your users and your API providers. Quite literally, I'm already thinking about new domains this topology can be applied to.

Overall, I'm excited to see what other apps & patterns emerge as developers consider 'staying off the server'.

Stream Audio To Your Home Theater Without Using Airplay

I wrote a post earlier this week about live streaming system audio from Mac OSX.

My home theater is powered by a Denon AVR-1912. One reason I chose this receiver is that supports Airplay directly. Over the last year it's become well-integrated with the rest of my chain. I can control it via the Denon Remote iPhone App or access it via a web interface:

Denon web interface

Because it has an HTTP interface, we can reverse-engineered wire traffic and build an API for it. You can learn a lot creating API's this way and it's especially gratifying if the resulting integration becomes useful to you or someone else.

After a quick glance I can tell it'll be a snap to plug this Denon into party.io's 'Rabot' so it can control not only the playlist stream but the actual Denon client playing the music. For example, to turn up the bass:

rabot prepare to DROP  #=> POST /ChannelLevel.asp?_subwoofer_offset=+10

Airplay vs. Streaming server

The Mac OSX setup I wrote up still lets you send audio to Airplay targets - a huge bonus. But, then the audio ceases to reach line out. You can't have both without more trickery, which to the perfectionist is clearly not acceptable. (And keeping in mind Windows users for whom Airport isn't an option.)

The ideal solution is to have the Denon consume the Icecast streaming audio feed, just like all of the other devices around the company.

Fortunately, this Denon has an Internet streaming radio capability. Unfortunately, I couldn't find anywhere to enter a custom URL for a radio station. Things weren't looking good - but this only bolsters the will.

Enter vTuner

On the Internet Radio options list there's an option labeled "radiodenon.com". It turns out this is a 3rd party integration with a platform called vTuner - "The vTuner platform connects your Internet enabled product to the wide world of streamed music, talk, and video." vTuner provides a logged-in user experience where you can add various radio stations as favorites. The favorites then appear on your Denon automagically.

For this to work, vTuner needs to support creating a new station that's not already in their directory. Thankfully it does!

How to configure vTuner + Denon

  • Find your Denon's MAC address per the intructions here. Or just look at your router's DHCP table.
  • Register for a Denon vTuner account at www.radiodenon.com. This process is a little wonky. You'll actually first sign in with the MAC Address of your Denon.

    vTuner Register

    Which brings me to:

    Mac Address

  • Once you've created an account and logged in, choose the "Add Another Station" option by clicking the nearby Go button. You'll be taken to this screen:

    vTuner Add Station

  • Enter the required fields and click 'Go'. The station URL is the URL to your streaming Icecast instance.

  • Now that your station is added, your Denon should have it on the list. We'll use the web interface to check. To enter the web interface, just point a browser to your Denon's IP address. You can find the IP by looking at system info on the unit or by looking at your router's client table.

  • In the web interface, navigate to NET Audio/USB via the right-side boxes. Then choose Internet Radio. Then click the radiodenon.com option.

    Denon Internet Radio

  • You should be taken to this page (with your unique ID showing):

    Denon radiodenon.com

  • Click the ID# option and you should see the radio station you created on vTuner - ours is called Spin Party. Click it to connect and start playing!

    Denon playing vTuner station

Success

So there you have it - streaming system audio to your receiver or any of your Internet-savvy devices without requiring Airplay.

Live-stream system audio from OSX Mountain Lion with Icecast2 and Darkice

Any team can benefit from a widely-accessible streaming music server. Sharing the same music increases team flow and builds culture. And funny things happen.

To share both music and announcements, party.io uses Play from Github. Play is an 'employee-powered iTunes-based client-driven distributed music server for your office'. Our company DJ (a big-hearted, chrome-plated hubot) manages Play and occasionally accepts submissions from the team.

Play syncs the company song queue, but a streaming audio server is still needed to stream what's playing. There are products that do this; Nicecast is one.

You can also build an open-source frankenstack for free. It's just four pieces: Soundflower, Jack, Darkice and Icecast. This post explains how to put these pieces together.

While it's a little more work, the outcome is the ability to stream system audio anywhere. This makes more scenarios possible and it requires less maintenance - there's no requirement to add media files in advance or to configure any applications.

If you can hear it, you can stream it.

Before starting

You'll need the latest compiler installed, whether via XCode or the OSX Mountain Lion Command Line Tools installation at http://connect.apple.com/ (recommended if you don't need all of XCode). You'll need to be somewhat comfortable with working on the command line. Make sure you have Homebrew installed. And make sure to run brew update before beginning, and brew doctor if anything goes wrong along the way.

Installation: Soundflower

Soundflower is an inter-application audio routing utility. Soundflower creates additional audio devices that bind to applications to send and receive audio. Soundflower also comes with a program called Soundflowerbed. It lets you listen to audio locally even while broadcasting, like a splitter. Here's what the Soundflowerbed menu bar app looks like:

Soundflowerbed

Download Soundflower here and install it. Homebrew doesn't have a Soundflower formula AFAIK, but thankfully there's nothing tricky to this installation.

Once you've restarted we'll install the next component, Jack.

Installation: Jack

Jack is a professional quality audio tool that can also route audio between applications.

Jack had problems initially with Mac OSX 10.7.3+, but a beta version seems to have worked the bugs out. Neither the Homebrew formula nor the latest download on jackosx.com have the new code, but you can find it on the Jack-OSX Yahoo group.

There's a link to the 0.90b8 beta package on this thread. Download and install it, restarting when necessary.

Next, start the Soundflowerbed app (in /Applications/Soundflower) and choose the 'Audio Setup' option from the menu bar icon it installs. This will launch the System Audio Midi Setup menu. Set both input and output to Soundflower (2ch), like in the image below. Also make sure the volume is up.

Audio Midi Setup

Now we can start Jack as follows:

➜  ~  jackdmp -d coreaudio
jackdmp 1.9.9
Copyright 2001-2005 Paul Davis and others.
Copyright 2004-2012 Grame.
jackdmp comes with ABSOLUTELY NO WARRANTY
This is free software, and you are welcome to redistribute it
under certain conditions; see the file COPYING for details
JACK server starting in realtime mode with priority 10
Input channel = 0 ==> JACK input port = 0
Input channel = 1 ==> JACK input port = 1
JACK output port = 0 ==> output channel = 0
JACK output port = 1 ==> output channel = 1
CoreAudio driver is running...

Jack will now be running and can pass audio from Soundflower into our next component, Darkice.

Installation: Darkice

Darkice is a live audio streamer. In our setup, it's responsible for encoding the audio from Jack (into MP3, Ogg Vorbis, et. al.) and sending it to Icecast.

We'll install Darkice with Homebrew but this gets a little weird. Darkice has a dependency on Jack, but Homebrew doesn't know we've already installed Jack through another means. So we have to install a 'dummy', unlinked Jack so Homebrew thinks the Jack dependency is satisfied but Darkice builds against the downloaded Jack and not the Homebrew Jack (this is essential).

Run this command:

brew install darkice --use-llvm   #ignore the linking warnings for Jack - Darkice must complete without Jack linked!

During this process Jack will fail to link and issue a warning but the script will still continue. This is good, because Darkice will then build against the downloaded Jack. If things blow up which installing dependencies, install without the llvm flag and then retry this again. The goal is just to get the Jack-failed-link and the Darkice build in the same step.

Installation: Icecast

Icecast is the web-serving layer of our stack, accepting incoming client connections and then broadacsting out the stream it receives from Darkice.

Install Icecast with Homebrew.

brew install icecast

Running Icecast and Darkice

We need a config file for Icecast: download this sample icecast.xml I've put together for this tutorial into ~/.icecast.xml. Now we can start Icecast:

icecast -c ~/.icecast.xml

We also need a config file for darkice: download this sample darkice.cfg into ~/.darkice.cfg. Once you have it saved, let's start Darkice:

darkice -c ~/.darkice.cfg

We can now check our status by logging into Icecast's administration UI. Go to http://localhost:8000/admin/ and enter the admin-user and admin-password from the icecast.xml config file. You should arrive at the admin page:

Icecast Admin

And you should see an area below that shows the active mount point, created by Darkice:

Icecast with Darkice mount

Testing with a client

Now, head over to another computer on your network, or your smartphone, and head for to the streaming address we've set up. The URL should look something like this:

http://computer.local:8000/spin

Substitute 'computer.local' for the hostname or IP address of your computer. 'spin' is the name of the mountpoint we specified in darkice.cfg (and 8000 the port).

Paste the URL into Safari on Mac or iOS and you should get playback in iTunes / iOS Quicktime respectively:

Streaming from iTunes Streaming on iPhone

Now you, your team, or the whole Internet can listen to your music, broadcasts, and the output of the command say! Flex the golden pipes!

Configure your router

You won't be able to access your streaming server from outside your home network unless your open the appropriate ports on your router. In this example, you'd open port 8000. Your router's instruction manual can you tell you how.

Bonus Gist

If you get it all to work, you might want to run these programs in the background and on startup. I wrote up a few basic plist files to be used with launchd that do just that. You can find them in this gist.

Credits

The creators of all of these fine tools. This article: Broadcasting music to multiple speakers.

Perception of Creativity in Development

Previously I talked about the concept of flow.

In this post I'll talk about why, specifically for software developers, flow conditions remain painfully absent from today's working environments. (1)

Like so many things it all comes down to perception. That's where we'll start.

Developers are sexy (it's just hard to tell sometimes)

Let's face it - from the observer's point of view, developers sit at a keyboard and type.

Observers can watch painters paint. Observers can watch architects draw and chefs slice and dice. But they only can watch developers… type. Typing is just not that sexy - or revealing. But ostensibly that's what developers do.

"But what about authors (novelists)? They're also ostensibly 'typing' - but they're sexy!" Authors are 'sexier' than developers because the finished product still resembles the "source code" - the thoughts from which it came - and it's in a shared language, about familiar topics, and still enjoyable atop its abstractions. The readers get a window to the author's soul, and a human connection is formed as the reader climbs through it. (2)

Using a developer's software isn't equivalent to reading a writer's thoughts - the human connection is far less apparent. Readers can imagine and romanticize about an author's creative process in a way that software users largely cannot. Software requires a far greater abstraction to create desirable and usable things.

Managers and users

The creative process behind software development can be as opaque to managers as it is to users. A big reason we don't see a lot of flow in the workplace is because it takes management support to bring about change - and managers don't fully understand the creative challenges developers face and why our work requires so much imagination.

This isn't to blame managers - the developer's process, as I mention, is largely mysterious to the observer. It's not hard to understand why coding doesn't look or seem all that glorious in the way art, design or physical science does. But it is :) The responsibility lies with the developer to communicate that - he or she needs to open the dialogue.

Yet I worry that developers are still commonly seen as builders - not creators. And it's hard to make a case for creative-level support if you're seen as just a builder.

The (dreaded) annual review

Developers are notoriously hard to review and compensate accurately per their skill level. This is also because observation is hard. And the ensuing confusion and tension is another reason why flow-like workplaces seldom materialize for developers.

Here's an example: Our example developer comes in late, is moody and highly opinionated, and has headphones on nearly 100% of the day, talking to no one. (3) Notwithstanding, the other developers still come to him or her with questions, and he or she personally tackles the heavy architectural challenges that come up.

How does this developer's review go? In progressive tech companies it might go well. Maybe peers will have a say like in Google's process. But in other environments it might end poorly. The reviewer might not be aware of the engineer's less visible, yet highly valuable, technical contributions. Another developer on the team might be sleeping under their desk every night, right after they break the build. A big raise for their 'loyalty' may be granted, but often to the chagrin of the rest of the team.

It's a hard problem. How can a manager who doesn't write code gauge who's really doing what? Classics have been written about this. Menacingly, traditional measures like punctuality and attitude don't correlate well with contribution or intelligence, nor do programming 'metrics' like lines of code, # of commits, etc.

But… what about mind reading?

Bending the mind - Inception

If only the manager had a dream-exploring machine - like the one Leo uses in Inception to interview Ellen Page. During the interview she architects dream mazes Leo can see and participate in. Consequently, he can evaluate her ability first-hand. Sadly, this machine's not invented. Yet.

Until it is, (4) my advice to developers is to work hard at explaining what you do, and allow your managers and colleagues insight into your thought process. And if you're a developer who really needs flow time to get the most from that process, ask for it.

My advice to managers is to observe developers in their natural habitat - sharing GIFs in chat rooms - without being intrusive. Find and reward and find the leaders. Look out for developers that are talented but not hitting their potential. Give them creative work, space to do it in, and then see what happens.

Summary

The difficulty of accurately observing the development process makes another very critical, very personal process - assessment and compensation - that much harder.

Together these challenges, along with baggage built up from years of tension and misunderstanding around them, help illustrate why managers and developers have yet to come together and change the workplace into something more amenable to the developer's process. The way forward requires developers to speak up and explain their process. It requires managers to listen and respond - and to take on the creative task themselves of restructuring work to maximize output from developers working in peak conditions.



Footnotes
  1. If you've only worked at startups or in Silicon Valley you might think that developers are highly respected creators the world over, endlessly pampered with free food, high salaries, lavish environments, open-ended time et. al. But, um, most developers still get treated like a commodity at small and large companies. And it's for those companies I call attention to this matter.
  2. This isn't a perfect analogy. A writer's finished book is still different than the 'source code' - it's often been edited in some way, compiled from notes, outlines, experiences, etc.
  3. e.g. me. jk. ok sometimes.
  4. …and putting all the other wildly creepy implications aside…

Here's To Flow

Do you know about Csíkszentmihályi's concept of flow?

Flow is the mental state of operation in which a person performing an activity is fully immersed in a feeling of energized focus, full involvement, and enjoyment in the process of the activity.

You might recognize the 'flow state' by its many aliases - in the zone, on a roll, dialed in, in the groove. You can flow for business or pleasure, but it's no secret that the creative working population relies heavily on flow to produce high output consistently.

It's also becoming widely acknowledged that the modern day work environment sabotages the flow state. These are the requirements to flow:

  • Long stretches of uninterrupted time
  • A problem to solve that you truly care about
  • A skill level that lets you focus on the task more than the tools
  • Absence of blocking, external dependencies

How does the modern day workplace stack up? Not well. A recent study from Adobe analyzes creativity globally and tells us we're spending only 25% of our time at work creating, and that only 1 in 4 of us believe we're living up to our creative potential.

The concept of flow isn't universal yet, but it could be someday. The now-ubiquitous word stress did not enter the American business vernacular until 1956. Employees back then needed a way to describe the effect of unsafe and unfair working conditions. Creatives today need a way to describe the impact of flow-impeding workplaces on their productivity.