I’ll admit - until I gained a bit of experience with Webhooks, they were pretty confusing. They’re prevalent nowadays, so forming a strong understanding is worth the effort!
The first time I used webhooks, I was implementing a Discord notification system for OpenRSC. The notification system would tell players in a Discord channel that an item had been put up for auction.
Sadly I think the Open RSC team has sunsetted the functionality. Can’t say I blame them, because the auction updates got kinda spammy.
I remember what felt like a nebulous workflow of coordinating with the Discord channel admin to enable Webhooks, obtaining a magic URL to call, etc. The steps are captured here:
Some Good Resources
TODO populate this section!
Design Patterns
TODO populate this section!
Other Links
TODO populate this section!
Unorganized wall of text
https://github.com/realadeel/awesome-webhooks
https://hookdeck.com/blog/post/dealing-with-webhooks-sucks-but-theres-something-you-can-do-about-it
https://shortcut.com/blog/more-reliable-webhooks-with-queues
Subscription Webhooks
To comply with the REST Hooks pattern, there is a minimum set of pieces you’ll need to add to have a working implementation. This article will outline those minimums:
Mechanism to store subscriptions Mechanism to modify subscriptions via API List and implement event types Mechanism to send hooks
Case study: Wufoo is a popular survey product service which integrates with Zapier. Wufoo supports both classic webhooks (the user must copy and paste a URL from one app to the other) and REST Hooks (the webhook is automatically created).
Webhooks are an architectural pattern that enable developers to receive updates to data as they happen rather than polling for the latest updates.
Your application exists within a network of other applications. Since changes to your user data affects many others, you need a robust model that takes this reality into account. An event-driven approach ensures you can send the right notification to the right application at the time it occurs.
Polling is bad for everyone. It’s taxing to servers on both sides, which spend most of their time and resources with requesting responses that contain nothing new. When there is data, it’s only as new as the polling interval, which means it might not even satisfy a user’s thirst for instant data access. Other than webhooks, the only way to have up-to-the-second access to new data is to poll every second. While some developers may try that, it’s not likely to be sustainable.
- Create your subscription endpoint
Before sending notifications as webhooks, you need to know what to send notifications about. That’s where a subscription endpoint comes in handy. Developers will “subscribe” to certain actions and tell you where to send notifications.
Your app might store contacts. For your API, the endpoint would be something like /contacts. You’ll make another endpoint for webhooks, such as /webhook, and treat it like any resource for your API.
action - description
- GET /webhook List existing webhooks
- POST /webhook Subscribe to a new webhook with details sent in JSON body
- GET /webhook/{id} Retrieve details of an existing webhook
- PUT /webhook/{id} Update an existing webhook with details sent in JSON body
- DELETE /webhook/{id} Unsubscribe from an existing webhook
Your /webhook endpoint would likely have the actions listed above, which would allow you to list, create, read, update, and delete webhooks.
- Implement your webhook queue
Now that you have a way for developers to subscribe to webhooks and update settings, you’re ready to start sending notifications when the criteria matches. But when should you check the criteria and send the notifications? There are a few approaches you can take, each with its own advantages, complexity, and scalability.
Do it inline: This is the simplest and most naive approach and should be reserved for prototypes or test environments. When an event occurs that might activate a webhook (such as a user adds a new contact), check whether a webhook fits the criteria. For all webhooks that do match, fire off a connection to the URL on file. The questionable part, for scaling, is making the request inline while the end user waits. A better solution is to create a queue so you can better deal with timeouts and retries.
Create a DB queue: When you send webhooks inline, all your code is in one place. With this option, you’ll be creating a record in your existing database for each notification you need to send. Rather than initiating a connection immediately, you’ll need a separate process to frequently look for new notifications and send them. Since the end user isn’t waiting, timeouts are not as big of a concern. And if you need to retry a notification, just keep it marked active until it succeeds or is called the maximum number of times.
Use a proper queue: If you know you’ll need to scale your solution, use a tool specifically designed for that. You can use open source scalable queueing solutions like RabbitMQ or a service like Amazon Simple Queuing Service. This way, your interaction is limited to adding and removing “messages,” which tell you what webhooks to call. Like the DB queue, you need a separate process to consume items from the queue and send notifications. In addition to using a tool designed for this purpose, a proper queue also saves database resources for what it does best–providing data to your primary application.
Webhooks
Original post - Progrium / Jeff Lindsay
Web hooks are essentially user defined callbacks made with HTTP POST. To support web hooks, you allow the user to specify a URL where your application will post to and on what events. Now your application is pushing data out wherever your users want.
https://thenewstack.io/webhooks-the-building-blocks-of-an-event-driven-architecture/
A webhook programmatically provides a notification when something happens on a platform or service.
https://developers.asana.com/docs/overview-of-webhooks
Webhook is a technology implementation for PubSub over HTTP which makes Webhook technology a subset of PubSub. Apart from Webhooks, PubSub could user other means of subscription and publishing (e.g Email).
https://codeopinion.com/building-a-webhooks-system-with-event-driven-architecture/
Do you need to integrate with external systems or services? Leveraging an event driven architecture enables you to build a webhooks system that can be decoupled from your main application code. Enabling you to call external systems that have subscribed via webhooks in complete isolation from your application code.
Publish-Subscribe: We can leverage event driven architecture to decouple the concerns of saving the order and doing our webhooks integrations.
Webhooks: Using an event driven architecture and messaging can facilitate building a webhooks system that can be very robust, fault-tolerant, resilient, and decoupled from your primary application code.
There are two ways your apps can communicate with each other to share information: polling and webhooks.
Webhooks typically are used to connect two different applications. When an event happens on the trigger application, it serializes data about that event and sends it to a webhook URL from the action application—the one you want to do something based on the data from the first application.
https://snipcart.com/blog/what-are-webhooks-explained-example
You can register a webhook by registering the URL to notify once given events occur. This first step is usually done via either a UI or by API.
The route created holds the logic to be executed once the event occurs.
This way, a system doesn’t have to know the nature of what needs to be executed, it only needs to keep track of the routes to notify.
https://medium.com/prospa-engineering/webhooks-done-right-676d4e74578a
Webhook is based on the same concept. It is an event-driven approach that ensures a service provider can send the right notification to the right external services at the time it occurs. In another way, it is a machine to machine architectural pattern that enables a client API to receive updates to data as they happen, rather than polling for the latest updates. Webhooks also have been known as reverse APIs as a server calls the client’s API. With most APIs there’s a request followed by a response. No request is required for a webhook.
Webhooks - security
http://resthooks.org/docs/security/
Shared Signals and Events – A Secure Webhooks Framework The Shared Signals and Events (SSE) Framework improves API efficiency and security by providing privacy-protected, secure webhooks. It is in use by some of the largest cloud services to communicate security alerts and status changes of users, continuously and securely to prevent and mitigate security breaches. It is currently leveraged by two applications – the Continuous Access Evaluation Protocol (CAEP) and Risk Incident Sharing and Coordination (RISC) to achieve this result.
https://zapier.com/blog/what-are-webhooks/
And they’re typically secured through obscurity—each user of an application gets a unique, random URL to send webhook data to—though they can optionally be secured with a key or signature.
** Authenticated+Authorized Webhooks - User A cannot send notification to User B webhook
Webhooks - examples
- https://developers.asana.com/docs/get-multiple-webhooks
- https://developers.asana.com/docs/example-webhook-servers
- https://github.com/Asana/devrel-examples/tree/master/javascript/webhooks-nodejs
- https://stripe.com/docs/webhooks
Webhooks - design & patterns
- http://resthooks.org/
- http://resthooks.org/docs/alternatives/
- https://zapier.com/engineering/webhook-design/
- https://thenewstack.io/webhooks-the-building-blocks-of-an-event-driven-architecture/
- https://web.archive.org/web/20120302000920/http://wiki.webhooks.org/w/page/13385128/RESTful%20WebHooks
Webhooks - Static webhooks vs subscription webhooks
https://zapier.com/engineering/webhook-design/
Note: You may be familiar with adding a single webhook URL into an admin interface when working with other APIs. That is called a static webhook and is not recommended for robust subscription notifications. Static webhooks can only notify one URL and are limited to a single set of criteria. Plus, a user needs to manually add every notification they want. By comparison, subscription webhooks allow a single application to receive multiple types of notifications. Each can be added via API so a user is not required to interact with each subscription set up.
https://zapier.com/engineering/static-webhooks/
By contrast, subscription webhooks provide an API endpoint through which applications can register for notifications. Using the same authentication as other API calls, a developer can create new subscriptions to events. Well-designed subscription webhooks allow for any number of subscriptions, each declared for just the data that is needed. Because they’re part of the API, subscriptions webhooks can also be automated, which saves users from being the intermediary when setting up webhooks.
Case study: Wufoo is a popular survey product service which integrates with Zapier. Wufoo supports both classic webhooks (the user must copy and paste a URL from one app to the other) and REST Hooks (the webhook is automatically created).
- https://docs.workato.com/developing-connectors/sdk/guides/building-triggers/static-webhook.html
- https://docs.workato.com/developing-connectors/sdk/guides/building-triggers/dynamic-webhook.html
Webhooks (push) vs Polling (pull)
Machine-machine architecture pattern, that enables receiving updates
Webhooks are comm’s for external systems, vs Message Queues for internal systems
“Implementing webhooks” = give me the URL to notify
data from an API delivered proactively