Category Archives: HTTP Concepts

HTTP Long Polling

Web applications were originally developed around a client/server model, where the Web client is always the initiator of transactions, requesting data from the server. Thus, there was no mechanism for the server to independently send, or push, data to the client without the client first making a request.

To overcome this deficiency, Web app developers can implement a technique called HTTP long polling, where the client polls the server requesting new information.  The server holds the request open until new data is available. Once available, the server responds and sends the new information. When the client receives the new information, it immediately sends another request, and the operation is repeated. This effectively emulates a server push feature. 

Considerations for HTTP Long Polling

There are a couple of things to consider when using HTTP long polling to build realtime interactivity in your application, both in terms of developing and operations/scaling.

  • As usage grows, how will you orchestrate your realtime backend?
  • When mobile devices rapidly switch between WiFi and cellular networks or lose connections, and the IP address changes, does long polling automatically re-establish connections?
  • With long polling, can you manage the message queue and catch up missed messages?
  • Does long polling provide load balancing or failover support across multiple servers?

When building a realtime application with HTTP long polling for server push, you’ll have to develop your own communication management system. This means that you’ll be responsible for updating, maintaining, and scaling your backend infrastructure.

Backend Infrastructure for Realtime Functionality

With these considerations in mind, that’s where a realtime data stream network comes in. This data stream network takes care of the backend infrastructure for you, so you don’t have to worry about maintaining and orchestrating your realtime network.

When looking at HTTP long polling with the goal of streaming data, PubNub is a low-latency and low-overhead realtime Web app communication environment, and features the ability to send messages to a single client, client groups and all clients. Upgrading to PubNub is both rapid and easy since it is based on a publish/subscribe model.

Key Benefits of Protocol Agnostic Realtime Messaging

Instead of relying solely on HTTP long polling for realtime messaging, a protocol-agnostic approach is beneficial. PubNub automatically chooses the best protocols and frameworks depending on the environment, latency, etc. Any server or client code wanting to communicate makes a single API call to publish or subscribe to data channels.

The code is identical between clients and servers, making implementation much simpler than using HTTP long polling. In terms of connection management, the network handles network redundancy, routing topology, loading balancing, failover, and channel access. Additionally, it offers core building blocks for building realtime into your application.

  • Presence – Detect when users enter/leave your app and whether machines are online
  • Storage & Playback – Store realtime message streams for future retrieval and playback
  • Mobile Push Gateway – Manage the complexities of realtime apps on mobile devices, including Push Notifications
  • Access Management – Fine grain Publish and Subscribe permissions down to the person, device or channel
  • Security – Secure all communications with enterprise-grade encryption standards
  • Analytics – Leverage visualizations into your realtime data streams
  • Data Sync – Sync application state across clients in realtime
'Coz sharing is caring

SPF, DKIM, AND DMARC

Emails consist of various part and each part focuses on certain protocol.

SPF

SPF focuses on the “domain” found in an email header that has many names, Return-Path, MAIL-FROM, Bounce address, Envelope from. In this article, we will refer to it as the Return-Path. If this header is missing, SPF falls back and looks at the “HELO/EHLO” hostname and checks for an SPF record there.

Don’t forget — the Return-Path header is a technical header that is not visible to the end user.  Unless they know how to display the headers of an email in their mail client they won’t see it. 

DKIM

DKIM focuses on the “DKIM-Signature” header.

Don’t forget — this header again is not visible to the end user unless they know how to display the headers of the email they have received.

DMARC

DMARC focuses on the domain found in the “From or Header from” header which is visible to the end user. In this article, we will refer to it as “From”.

Now that we know what headers each protocol looks at, what is actually contained in those headers and what is checked?

SPF —This verifies if an email was sent by an authorized sender by checking a list of authorized IP addresses you publish in your DNS. The receiving server will take the domain found in the Return-Path header and check for an existing SPF record. It checks the SPF record to see if the sending IP address of the email is actually contained in the SPF record. If the IP address is contained in the SPF record that means that it is authorized to send emails. This means that SPF PASSED. If the IP address is not in the SPF record then SPF FAILS.

The overall logic is:

If the sending IP address is contained in the SPF record = SPF PASS

If the sending IP address is not contained in the SPF record = SPF FAIL

DKIM — The receiving server will check the DKIM-Signature header which contains the selector (s=) and signing domain (d=) which are tags used to look up the public key. Once retrieved, the public key is used to validate the email message. If validation is successful then DKIM PASSES and if the validation process is unsuccessful then DKIM FAILS.

The overall logic is:

If validation is successful = DKIM PASS

If validation is unsuccessful = DKIM FAIL

DMARC — The receiving server will check if either SPF or DKIM PASSED, then it will check if the Return-Path domain used by SPF and/or the “d=” domain used by DKIM align with “From” domain, and finally it will extract the DMARC policy published by the domain found in the “From” address and comply with the policy.

The overall logic is:

If SPF PASSED and ALIGNED with the “From” domain = DMARC PASS, or

If DKIM PASSED and ALIGNED with the “From” domain = DMARC PASS

If both SPF and DKIM FAILED = DMARC FAIL

DMARC not only requires that SPF or DKIM PASS, but it also requires the domains used by either one of those two protocols to ALIGN with the domain found in the “From” address. Only then will DMARC PASS.

What’s the difference between Strict vs Relaxed alignment?

Strict alignment means that the Return-Path domain or the signing domain “d=” must be an exact match with the domain in the “From” address.

Relaxed alignment means that the Return-Path domain or the signing domain “d=” can be a subdomain of the “From” address and vice versa.

What happens if DMARC fails?

If DMARC fails then the receiving server would typically comply with the policy that you have specified in your DMARC record.

If you are in report-only mode (p=none) the email will be accepted by the receiving server and scanned by other filtering criteria.

If you are in quarantine mode (p=quarantine) the email will be quarantined and typically sent to the spam folder of the recipient.

If you are in reject mode (p=reject) the receiving server will abort the connection with the sending mail server and the email will never reach the end user.

Irrespective of the policy, the metadata for the email will be logged along with the status of the authentication results and forwarded to your DMARC report processor.

SPF troubleshooting and top tips

  1. Make sure that you have an SPF record in your Return-Path domain.
  2. Make sure that you have an SPF record in your HELO/EHLO domain in case of bounces where the Return-Path domain is empty.
  3. Make sure there is a single SPF record per domain.
  4. Make sure that the SPF record syntax is correct.
  5. Make sure that your Return-Path domain aligns with the From domain.
  6. Make sure that your authorized senders are part of the SPF record.
  7. Make sure that unauthorized senders are not in your SPF record.
  8. Make sure that you do not go over the 10 DNS lookup limit imposed by SPF.
  9. Make sure that deprecated SPF record mechanisms such as the “ptr” mechanism are not used in your SPF record.

DKIM troubleshooting and top tips

  1. Make sure that the sending systems you use support DKIM.
  2. Make sure that the emails are DKIM signed.
  3. Make sure that the signing domain aligns with the “From” domain.
  4. Make sure that you use a DKIM key size over 1024 bits (a 2048 bit key is advisable)
  5. Make sure, where possible, that the DKIM selectors you choose closely identify the sending service so you can distinguish between them.
  6. Make sure to revoke any keys that have been compromised.
  7. Make sure that the DKIM keys you manage are rotated on regular basis.
  8. Make sure that the DKIM key syntax is correct.
  9. Make sure that there exists a public key for each corresponding private key that signs your emails.

DMARC troubleshooting and top tips

  1. As DMARC is based on both SPF and DKIM and the domains used by those two protocols, you will have to make sure that the Return-Path domain for SPF is either an exact match or a subdomain of the “From” domain. The same applies to the signing domain used by DKIM.
  2. Make sure that the DMARC record syntax is correct.
  3. Make sure that you have configured all of your systems correctly with SPF and DKIM before moving to a reject policy as your emails will be lost.
  4. Make sure that you use a system or third-party provider to receive DMARC reports so that you can make sense of those reports and discover any systems that are misconfigured.
  5. Monitor the status of each of your sending sources and make sure that any changes to SPF and DKIM are identified.
'Coz sharing is caring