Blog by Pulkit Goyal

An Introduction to Exceptions in Elixir

Tue, 01 Oct 2024 05:58:38 GMT

Exceptions are a core aspect of programming, and a way to signal when something goes wrong with a program. An exception could result from a simple error, or your program might crash because of underlying constraints. Exceptions are not necessarily bad, though - they are fundamental to any working application.

Let’s see what our options are for handling exceptions in Elixir.

Raising Exceptions in Elixir

The Elixir (and Erlang) community is generally quite amenable to exceptions: “let it crash” is a common phrase. This is due, in part, to the excellent OTP primitives in Erlang/Elixir. The OTP primitives allow us to create supervisors that manage and restart processes (or a group of related processes) on failure.

Exceptions can occur when something unexpected happens in your Elixir application. For example, division by zero raises an ArithmeticError.

iex> 1 / 0
** (ArithmeticError) bad argument in arithmetic expression: 1 / 0
    :erlang./(1, 0)
    iex:1: (file)

Exceptions can also be raised manually:

iex> raise "BOOM"
** (RuntimeError) BOOM
    iex:1: (file)

By default, raise creates a RuntimeError. You can also raise other errors by using raise/2:

defmodule Math do
	def div(a, b) do
	  if b == 0, do: raise ArgumentError, message: "cannot divide by zero"
	  a / b
	end
end

iex> Math.div(1, 0)
** (ArgumentError) cannot divide by zero
    iex:4: Math.div/2
    iex:3: (file)

A function call that doesn’t match a defined function raises an ArgumentError by default:

defmodule Math do
  def div(a, b) when b != 0 do
    a / b
  end
end

iex> Math.div(1, 0)
** (FunctionClauseError) no function clause matching in Math.div/2

    The following arguments were given to Math.div/2:

        # 1
        1

        # 2
        0

Handling Elixir Exceptions

Elixir provides the try-rescue construct to handle exceptions:

try do
  raise "foo"
rescue
  e in RuntimeError -> IO.inspect(e)
end

This prints %RuntimeError{message: "foo"} in the console but doesn’t crash anything. Use this when you want to recover from exceptions in Elixir. It is also possible to skip binding the variable if it is unnecessary. For example:

try do
  raise "foo"
rescue
  RuntimeError -> IO.puts("something bad happened")
end

In both of the above examples, we only rescue RuntimeError. If there is another error, it will still raise an exception. To rescue all exceptions raised inside the try block, use the rescue without an error type, like this:

try do
  1 / 0
rescue
  e -> IO.inspect(e)
end

Running the above prints %ArithmeticError{message: "bad argument in arithmetic expression"}. If you want to perform different actions based on different exceptions, just add more clauses to the rescue branch.

try do
  1 / 0
rescue
  RuntimeError -> IO.puts("Runtime Error")
  ArithmeticError -> IO.puts("Arithmetic Error")
  _e -> IO.puts("Unknown error")
end

While rescuing all errors (without using a specific type) sounds tempting, it is a good practice to rely on specific exceptions because:

You can perform different recovery tasks for different exceptions.
It saves you from future breaking changes or programming errors going unnoticed.

For example, consider the below function:

defmodule Config do
  def get(file) do
    try do
  	  contents = File.read!(file)
      parse!(contents)
    rescue
      _e -> %{some: "default config"}
    end
  end
end

It reads and parses a config file, returning the result. When written, it uses a generic clause to rescue from missing file-related errors. Let’s say that a while later, the input file gets corrupted somehow (e.g., a simple missing } or an extra , in a JSON file), and now there are parsing errors. Our function will still work without raising any issues, and we will be left guessing why it doesn’t work even though there’s an existing file.

Another common practice in the Elixir community is to use ok/error tuples to signal errors instead of raising exceptions (for both internal and external libraries). So, instead of returning a simple result or raising an exception on an issue, a function in Elixir will return {:ok, result} on success and {:error, reason} on failure.

This means that most of the time, a case block will be preferable to try/rescue. For example, the above File.read! can be replaced by:

case File.read(file) do
  {:ok, contents} -> parse!(contents)
  {:error, reason} -> %{some: "default config"}
end

In practice, you should reach out for try/raise only in exceptional cases, never as a means of control flow. This is quite different from some other popular languages like Ruby or Java, where unexpected operations usually raise an error, then handle control flow for cases like non-existent files.

Re-raising Exceptions

Sometimes, we just want to know that there is an exception but not rescue from it - for example, to log an exception before allowing the process to crash or to wrap the exception into something more useful/understandable to the user.

This is where reraise can be helpful, as it preserves an exception’s existing stack trace:

try do
  1 / 0
rescue
  e ->
    IO.puts(Exception.format(:error, e, __STACKTRACE__))
    reraise e, __STACKTRACE__
end

Here, we use the __STACKTRACE__ to retrieve the original trace of the exception, including its origin and the full call stack.

In a real-world application, you might use this to report a metric/send exception to a third-party exception tracking service, or log something informative to a third-party logging service. For example, you might send telemetry data to AppSignal under these circumstances.

In addition, it is also common practice to have custom exceptions for cases where unexpected things happen in libraries. reraise/3 can be useful here:

defmodule DivisionByZeroError do
  defexception [:message]
end

defmodule Math do
  def div(a, b) do
    try do
      a / b
    rescue
      e in ArithmeticError ->
        reraise DivisionByZeroError, [message: e.message], __STACKTRACE__
    end
  end
end

Now using Math.div(1, 0) will raise a DivisionByZeroError instead of a generic ArithmeticError.

Rescue And Catch in Elixir

In Elixir, try/raise/rescue and try/throw/catch are different. raise and rescue are used for exception handling, whereas throw and catch are used for control flow.

A throw stops code execution (much like a return - but with the difference that it bubbles up until a catch is encountered). It is rarely needed and is only an escape hatch to be used when an API doesn’t provide an option to do something.

For example, let’s say that there’s an API that produces numbers and invokes a callback:

defmodule Producer do
  def produce(callback) do
    Enum.each((1..100) , fn x -> callback.(x) end)
  end
end

(This is just an example. Assume that with the real API, it is much more expensive to produce each number, and it produces an infinite list of numbers.)

We want to find the first number divisible by 13, since we don’t have any other apparent way of finding that number and stopping the production at that point. Let’s see how we can use throw/catch to do this:

try do
  Producer.produce(fn x ->
    if rem(x, 13) == 0 do
      throw(x)
    end
  end)
catch
  x -> IO.puts("First number divisible by 13 is #{x}")
end

It’s worth stating that this is a bit of a contrived case, and most good APIs are designed in such a way that you never need to reach out for throw/catch.

After and Else Blocks in Elixir

You can use after and else blocks with all try blocks. An after block is called after processing all other blocks related to the try, regardless of whether any of the blocks raised an error. This is useful for performing any required clean-up operations.

For example, the below code creates a new producer and makes some results. If there’s an error during production, it will raise an exception. But the after block ensures that the producer is still disposed of regardless.

producer = Producer.new
try do
	Producer.produce!(producer)
after
  Producer.dispose(producer)
end

On the other hand, an else block is called only if the try block is completed without raising an error. The else block receives the try block’s result. The return value from the else block is the final return value of the try. For example:

try do
  File.read!("/path/to/file")
rescue
  File.Error -> :not_found
else
  "a" -> :a
  _other -> :other
end

In the above code, if the file exists with content a, the result will be :a. The result is :not_found if the file doesn’t exist, and:other if the content is anything else.

Exceptions and Processes

No discussion about Elixir exceptions is complete without examining their impact on processes. Any unhandled exceptions cause a process to exit.

With this in mind, let’s revisit the “let it crash” strategy. If we don’t handle an exception from a process, it will crash. This is good in a way because:

Since all processes are separate from each other, an exception or unhandled crash from one process can never affect the state of another process.
In most sophisticated Elixir applications, all the processes run under a supervision tree. So, an unexpected exit will restart the process (depending on the supervision strategy) and any linked processes it depends on with a clean slate.

In most cases, intermittent issues will resolve themselves in the next run. This is much easier (and usually also cleaner) than handling each failure separately and performing a recovery step.

If you want to learn more about supervisors, I suggest the official Elixir Supervisor and Application guide as a great starting point.

Monitoring Exceptions

As we’ve seen, exceptions are a fundamental part of any application. Handling them is good, but sometimes, letting them crash a process is even better. But in all cases, it is better to monitor exceptions happening in the real world so that you can take action if there’s something concerning (for example, a developer error that crashes an app).

This is where AppSignal for Elixir can help. Once set up, it automatically records all errors and can also trigger alerts based on predefined conditions

Here’s an example of an individual error you can get to from the “Errors” -> “Issue list” in AppSignal for debugging:

Check out the AppSignal for Elixir installation guide to get started.

Wrapping Up

In this post, we explored how errors are treated in Elixir and how to recover from them. We also saw that sometimes it is better to just let a process crash and be restarted through the supervisor than to manually perform recovery steps for all possible exceptions.

Elixir’s API makes a clear distinction between functions that raise an exception (usually ending in !) and functions that return a success/error tuple. If you are a library author, it is better to provide both options for users.

Reach out for the tuple-based methods when you need to handle error cases separately. In Elixir, we rarely use try blocks for control flow - the ! functions are for when we want a process to crash on unexpected events.

Until next time, happy coding!

An Introduction to Mocking Tools for Elixir

Wed, 07 Jun 2023 04:57:14 GMT

A well-written test suite is a big part of any successful application. But let’s say you rely on an external dependency for some parts of your app (for example, an external API for fetching user information). It then becomes important to mock that dependency in the test suite to prevent external API calls during testing or to test specific behavior.

Several frameworks help reduce the boilerplate and make mocking safe for Elixir tests. We will explore some of the major mocking tools available in this post. Let’s get started!

Test Without Mocking Tools in Elixir

Depending on the scope of your tests, you might not need to use any external mocking tools. You can use test stubs or roll your own server instead.

Test Stubs

The simplest way to stub/mock out some parts from a function call is to pass around modules that do the actual work and use a different implementation during the tests.

Let’s say you need to access the GitHub API to fetch a user’s profile with an implementation like this:

defmodule GithubAPI do
  def fetch_user(username) do
    case HTTPoison.get("https://api.github.com/users/#{username}") do
      {:ok, response} ->
        parse(response)
      {:error, reason} ->
        {:error, reason}
    end
  end
end

We know the implementation relies on HTTPoison to make actual calls to the GitHub API. To mock it during tests, we can update the implementation to pass around an http_client and use a different one during tests. Something like this:

defmodule GithubAPI do
  def fetch_user(username, http_client \\ HTTPoison) do
    case http_client.get("https://api.github.com/users/#{username}") do
      #... handle results
    end
  end
end

This approach aligns with the Law of Demeter - a module should only have a limited knowledge of the objects (in this case, the HTTP client) it works on. This approach is nice because it decouples the GithubAPI module from the HTTP client implementation. Our GithubAPI users can continue using it in the same way. But for unit-testing GithubAPI, we can pass in our custom http_client that returns just the results we need. For example:

defmodule GithubAPITest do
  defmodule GithubHTTPClientUser do
    def get("https://api.github.com/users/octocat") do
      {:ok, %HTTPoison.Response{status_code: 200, body: ~s<{"login": "octocat"}>}}
    end
    def get("https://api.github.com/users/unknown") do
      {:error, %HTTPoison.Error{message: "Not Found"}}
    end
  end

  test "can fetch a user" do
    {:ok, %Github.User{login: "octocat"}} = GithubAPI.fetch_user("octocat", GithubHTTPClientUser)
  end

  test "handles errors when fetching a user" do
    {:error, %HTTPoison.Error{}} = GithubAPI.fetch_user("unknown", GithubHTTPClientUser)
  end
end

That works, and you have 100% test coverage! 🎉 But as you can already imagine, while this works well for small tests, it will become increasingly difficult to manage if our GithubAPI class grows to include other features. Another similar strategy is to use application configuration instead of passing around the client modules. This is especially useful when you need to mock out the API from all the tests, even when this API is called from other internal modules. We can update our code to this:

# github_api.ex
defmodule GithubAPI do
  @http_client Application.compile_env(:my_app, GithubAPI, []) |> Keyword.get(:http_client, HTTPoison)

  def fetch_user(username) do
    case @http_client.get("https://api.github.com/users/#{username}") do
      #... handle results
    end
  end
end

# test/support/mock_github_http_client.ex
defmodule MockGithubHTTPClient do
    # ... All mock implementations here
end

# config/test.exs
config :my_app, GithubAPI, http_client: MockGithubHTTPClient

The good thing is that you don’t need to worry about mocking out GithubAPI calls in each test case by manually passing the HTTP client. This is especially important when the actual API calls are nested inside other functions. For example, if your application automatically fetches the user from GitHub after a new account is created, you don’t need to mock GithubAPI everywhere you create new users. But it still has the same disadvantages as the previous strategy. Plus, the mocks must be generic enough to be used throughout the test suite.

Roll Your Own Server

This one is interesting. Since plug and cowboy make it really easy to roll out an HTTP server, instead of mocking out the HTTP Client, we can start our own server during tests and respond with stubs instead.

If you want to learn more, check out:

In the strategies discussed above, quite a bit of boilerplate is involved in setting up the tests. And if you need a way to validate that a specific function was called with specific arguments, you need to do additional work. If you have just a small external dependency that you need to mock out, this might work well for you. But if there are complex edge cases and branches that you need to test out, mocking tools can help to simplify the complexity of writing and maintaining those mocks in the long run.

Elixir Mocking Tools

Mock

Mock is the first result you will see when searching “Elixir Mock”, and is a wrapper around Erlang’s meck that provides easy mocking macros for Elixir.

With Mock, you can:

Replace any module at will during tests to change return values.
Pass through to the original function.
Validate calls to the mocked functions.
Check the complete call history, including arguments and results for each call.

It is a very powerful tool, and the macro with_mock makes mocking during tests really easy. Let’s see how we can rewrite our test case to validate GithubAPI.fetch_user:

defmodule GithubAPITest do
  # This is important, Mock doesn't work with async tests!
  use ExUnit.Case, async: false

  import Mock

  test "can fetch a user" do
    with_mock HTTPoison, [get: fn _url -> {:ok, %{status_code: 200, body: ~s({"login": "octocat"})}} end] do
      {:ok, %Github.User{login: "octocat"}} = GithubAPI.fetch_user("octocat")
      assert_called HTTPotion.get("https://api.github.com/users/octocat")
    end
  end
end

Very sleek. There’s no need to define and pass boilerplate modules around or fiddle with the config just for tests. Our implementation is completely isolated from the test code and needs no special changes to fit the tests. And if we need to validate that a function was called with specific arguments, we can do that as well with assert_called. One of the major drawbacks of this strategy is that you cannot use it with async tests. It doesn’t prevent you from using Mock inside asynchronous tests, so this can lead to hard-to-track flaky tests. In my opinion, Mock works well for certain types of tests. I usually find myself reaching for it when we need to mock external libraries that we have no control over. For example, let’s say we use an external library to fetch a user’s GitHub profile instead of our custom GithubAPI. Something like this:

def create_user(github_username) do
  Tentacat.Users.find(Tentacat.Client.new(), github_username)
  |> case do
    {:ok, user} ->
      # ... create user
    {:error, error} ->
      # ... handle error
  end
end

Now, if we dig into the library’s documentation/code, we can see that it uses HTTPoison to make the eventual request to the API. But we don’t want to - or have a way to - customize that client just for tests. Here, with_mock can easily help mock out that call so that we don’t request the API. In fact, a much better approach, in this case, is to use with_mock to simply mock out the Tentacat.Users.find call altogether. That saves you from having to dig into the library’s internal code (which can change with any update) and relying solely on mocking out its public interface.

Mox

We saw above how easy it is to mock some methods out and have our tests pass. We sprinkle these calls in a few places to mock HTTPoison, and we are done. But what if we later decide that HTTPoison isn’t fast enough and want to switch to another HTTP client implementation? Suddenly, all our tests fail - we have to go back and fix them up. Even worse, what if the API for HTTPoison changes, but since we mocked it out, our tests never failed, and we pushed something that didn’t work to production?

Mox helps get around these issues by ensuring explicit contracts. Read Mocks and Explicit Contracts for more details. Using our GithubAPI example above, this is how we need to set up the tests with Mox.

Mock the External Client

We first convert our API client into a Behaviour to define the explicit contract the API client should follow.

# lib/my_app/github/api.ex
defmodule MyApp.GithubAPI do
  @callback fetch_user(String.t()) :: {:ok, Github.User.t()} | {:error, Github.Error.t()}

  def fetch_user(username), do: impl().fetch_user(username) do

  defp impl(), do: Application.get_env(:my_app, GithubAPI, GtihubAPI.HTTP)
end

Next, we define the API Client that makes the actual requests to fetch the user.

# lib/my_app/github/http.ex
defmodule MyApp.GithubAPI.HTTP do
  @behaviour MyApp.GithubAPI

  @impl true
  def fetch_user(username) do
    case HTTPoison.get("https://api.github.com/users/#{username}") do
      {:ok, response} ->
        # Parse response here...
        {:ok, user}
      {:error, error} ->
        # Handle error here...
        {:error, reason}
    end
  end
end

Finally, in our test helper, we define a mock MyApp.GithubAPI.Mock for the API and set it in our application environment.

# test/test_helper.exs
Mox.defmock(MyApp.GithubAPI.Mock, for: MyApp.GithubAPI)
Application.put_env(:my_app, GithubAPI, MyApp.GithubAPI.Mock)

Now we can refactor the test case to use Mox for mocking out the call to the external API:

# test/my_app/github/api_test.exs
defmodule GithubAPITest do
  use ExUnit.Case, async: true

  import Mox

  setup :verify_on_exit!

  test "can fetch a user" do
    MyApp.GithubAPI.Mock
    |> expect(:fetch_user, fn "octocat" -> {:ok, %Github.User{login: "octocat"}} end)
    assert {:ok, %Github.User{login: "octocat"}} = GithubAPI.fetch_user("octocat")
  end
end

There are several interesting aspects to the above code. Let’s break it down.

First, we use the expect function to define an expectation on the API. We expect a single call to fetch_user with the argument "octocat", and we mock it to return a {:ok, %Github.User{}} tuple. Now, when we call the GithubAPI.fetch_user/1 in the test, it will reach for that mocked expectation instead of the default implementation. Thus, we can safely assert the result of the call without making calls to the actual API. The verify_on_exit! function as setup ensures that all expectations defined with expect are fulfilled when a test case finishes. So if we define an expectation on fetch_user and it isn’t actually called (e.g., if the implementation changes later), we see the test fail. Finally, notice that we don’t need to mark the test as non-async here. That’s because Mox supports mocking inside async tests. So you can define two completely different mocks in two test cases, and they will both pass, even if they run simultaneously. You can also define a global stub to avoid providing mocks in every test. This is useful if your client is being used in several places and you don’t want to explicitly define and validate expectations everywhere. To do this, just update your ExUnit case template:

defmodule MyApp.Case do
  use ExUnit.CaseTemplate

  setup _context do
    Mox.stub_with(MyApp.GithubAPI.Mock, MyApp.GithubAPI.Stub)
  end
end

And create a stub with some static results:

defmodule MyApp.GithubAPI.Stub do
  @behaviour MyApp.GithubAPI

  @impl true
  def fetch_user("octocat"), do: {:ok, %Github.User{login: "octocat"}}
end

Now every time you use MyApp.Case in a test case, you don’t need to manually mock calls to GithubAPI - they will automatically be forwarded to the stubbed module. This works well when you have calls to the GithubAPI that you will hit across several test suites. With a stub, you can be sure that such calls always return a specific and stable response without having to mock them out manually.

Test the External Client

If you are following along closely, you will notice that we didn’t actually test the MyApp.Github.HTTP module at all. Don’t worry, we won’t leave it untested. But the recommended way here is to do integration tests instead of using mocking at that level. We will also configure it so that these tests don’t run when you run the full test suite.

# test/my_app/github/http_test.exs
defmodule MyApp.GithubAPI.HTTPTest do
  use ExUnit.Case, async: true

  # All tests will ping the API
  @moduletag :github_api

  # Write your tests here
end

# test/test_helper.exs
ExUnit.configure exclude: [:github_api]

The next step is to set up your CI pipeline to ensure that these tests run when required. For example, you can configure it to run on all pull requests targeting main to avoid reaching the external API on every commit. You can also check that your CI pipeline runs before anything is merged to the main branch. To do this, use the include command line flag when running mix test.

mix test --include github_api

There are several pros to the above strategy. You are now testing all parts of the application, including the calls to the external API (this part is not exactly dependent on Mox, but since we mocked out a significant part of our HTTP client, we must test it separately). We can also define global stubs, specific mocks, and expectations only when required, which makes most of our test- suite very clean and concise. The major drawback is that it requires quite some setup - you need a new behaviour with @callbacks for each public method you want to mock out. You have to implement that behaviour inside your module and finally set up Mox to mock out that implementation during tests.

And since we are now reaching the external API, the tests can be flaky, depending on the API’s availability.

Mimic

If you are used to Mocha for other languages, you can check out Mimic. It lets you define stubs and expectations during tests by keeping track of the stubbed module in an ETS table. It also maintains separate mocks for each process, so you can continue using async tests. It’s a great alternative to Mock - but that also means the same caveats apply - be careful about what you mock. Here’s what a sample test looks like with Mimic:

# test_helper.exs
Mmimc.copy(HTTPoison)
ExUnit.start()

# github_api_test.exs
defmodule GithubAPITest do
  use ExUnit.Case, async: true
  use Mimic

  test "can fetch a user" do
    url = "https://api.github.com/users/octocat"
    expect(HTTPoison, :get, fn ^url -> {:ok, %{status_code: 200, body: ~s({"login": "octocat"})}} end)
    assert {:ok, %Github.User{login: "octocat"}} = GithubAPI.fetch_user("octocat")
  end
end

expect/3 automatically verifies that the function is called. And all expect and stub calls can be chained, which makes up clear mocking code.

Use Mocking Right for Your Elixir App

Mocking is an important and necessary part of any test suite. But the thing to remember about mocking is that it is imperative to do it right.

For example, it might be tempting to mock out an internal API to quickly simulate something for a test. And yes, it works for quick unit tests. But then, someone else (or maybe you) comes along a while later and changes that something that you mocked earlier.

Your tests still pass since they use the mocked value. But in practice, the thing is now broken, and it will be caught much later in the feedback loop (or worse still, be shipped to the user as-is).

Wrap Up

In this post, we explored several mocking strategies you can use in Elixir tests. The safest one, and the one you should consider using first, is Mox since it forces mocked modules to have a defined behavior. Mox can therefore catch issues that arise from API changes during compilation.

Reach out for Mimic or Mock when you need to mock external libraries you don’t have any control over.

Until next time, happy mocking!

Database Performance Optimization and Scaling in Rails

Thu, 16 Feb 2023 06:46:10 GMT

Web applications usually rely heavily on databases, for the most part. And as applications grow, databases grow too. We keep scaling web servers and background workers to keep up with a heavy load. But eventually, the database needs to keep up with all the new connections from these processes.

One way to tackle this is to grow a database with an app using vertical scaling. This means adding more CPU power and memory to the database server. But this is usually slow. You might have to copy all your data to the new server and then put the application down to change the database servers that it communicates with.

This is also usually a one-way operation. You can’t keep adding/removing CPU power or memory from the database server based on your load.

This post will cover some alternative methods to fine-tune and scale a database under heavy load and improve the performance of your Rails app. We will focus on:

Splitting schema and data between multiple databases
Using read-only replica databases

Let’s get going!

Disclaimer: Scaling Isn’t Always Needed

As always with any post about performance optimization and scaling, I would like to put up a standard disclaimer: make sure you have an issue before trying to solve it.

For example, on one of our apps in production, we tackled scaling and optimization only when we started processing more than 3 million background jobs a day.

This will, of course, be different for different apps. But usually, a good indicator of whether you need to optimize is if your database is always running at high CPU or memory usage.

A Separate Database for Your Rails Application

Sometimes, simply using a separate database server for a part of your app is a good solution. For example, your app might do two different things that don’t have a big overlap. Alternatively, maybe there’s one very database-heavy feature, but it is used only rarely or by a small section of users. Go the way of a separate database server if - and only if - there is a clear distinction between the parts of your app that will use different databases.

An example of this being useful is audit logging in a high-frequency app (or other kinds of very high-volume data that isn’t necessarily accessed very frequently). Let’s see how we can set this up.

First, set up the database.yml to configure the second database. Let’s call the second database log. We’ll add a log_default entry for the secondary database with a common configuration across all environments:

log_default: &log_default
  host: localhost
  port: <%= ENV["DB_PORT"] || 5432 %>
  adapter: postgresql
  encoding: unicode
  user: <%= ENV["DB_USER"] || "postgres" %>
  password: <%= ENV["DB_PASSWORD"] || "postgres" %>
  migrations_paths: db/log_migrate

The important bit here is the migrations_paths option that tells Rails where to find migrations related to this database.

Then, update database.yml to configure access to the log database for development, test, and production environments:

development:
  primary:
    <<: *default
    database: <%= ENV["DB_NAME"] %>
  log:
    <<: *log_default
    database: <%= ENV["LOG_DB_NAME"] %>

test:
  primary:
    <<: *default
    database: my_app_test
  log:
    <<: *log_default
    database: my_app_log_test

production:
  primary:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_URL"] %>
  log:
    adapter: postgresql
    encoding: unicode
    migrations_paths: db/log_migrate
    url: <%= ENV["DATABASE_LOG_URL"] %>

As you can see, we move the configuration for each environment one level deeper. The configuration for our primary database lives inside the primary key for each environment. The second database’s configuration (that we call log) lives inside the log key. primary is a special keyword that tells Rails to use this database as the default.

The next step is to set up ActiveRecord to use this new database. We first create a base class that establishes a connection to this database (instead of a primary one) using establish_connection:

class LogRecord < ActiveRecord::Base
  self.abstract_class = true
  establish_connection :log
end

And then we inherit from LogRecord instead of ApplicationRecord for all our records that should access the log DB:

class AuditLog < LogRecord
end

Note that you keep using the rails generators for generating models and migrations by appending the --database log option to target the second database. The migration task now automatically migrates both databases, but if you need to only migrate one, you can use db:migrate:primary or db:migrate:log tasks.

This works great if you clearly distinguish between two parts of the app. But what if you don’t have a clear idea of the database that’s creating issues? There are still a couple of options left. Let’s check them out.

Using Read-Only Replicas for Rails

Our second scaling option is to access data from read-only replicas while still writing to the primary database. This can improve performance for users who only browse parts of the app without performing any writing operations. Depending on your app’s use case, this can be 80% of your users or 10% of them. So, evaluate the behavior of your users or your application use case before going down this route.

For an app with a majority of read-only users (like Twitter, for example), you can gain huge improvements in performance. New read-only replicas can be added/removed at will without affecting the primary database, which opens up options for auto-scalability.

Let’s see how to set this up.

Setting up a Read-Only Replica

As usual, we will start with modifications to the database.yml config to include the replica:

production:
  primary:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_URL"] %>
  primary_replica:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_REPLICA_URL"] %>

Again, the primary key is a special key indicating that this database is the default. We can use any other key for the replica database configuration. Let’s choose primary_replica for sanity.

Then update the ApplicationRecord to configure a connection to multiple databases:

class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true

  connects_to database: { writing: :primary, reading: :primary_replica } if Rails.env.production?
end

Note that we add configuration for a production environment only in the above example. If you want to, you can set it up for all environments, after modifying your development setup to support replicas.

Finally, to use the read-only replica, you need to tell Rails when to use the primary database and when to use the replica. Rails provides a basic implementation for automatic role-switching based on the HTTP verb of the request out of the box. To enable it, add the following to your production.rb:

# Use Read-Only Databases on GET requests
config.active_record.database_selector = { delay: 2.seconds }
config.active_record.database_resolver = ActiveRecord::Middleware::DatabaseSelector::Resolver
config.active_record.database_resolver_context = ActiveRecord::Middleware::DatabaseSelector::Resolver::Session

This tells Rails to use the replica for all GET or HEAD requests and the primary one otherwise. Notice the option that sets the delay to two seconds? That tells Rails to keep using the primary database for GET and HEAD requests if it is within two seconds of a write request in the same session.

Gotchas

Does all of this sound too simple? Well, we aren’t done yet. Depending on how your application is structured, you need to keep a few points in mind.

Firstly, ensure you do not write anything in a read-only action like index or show. If there are legitimate reasons for writing during those operations, you can manually switch the connection:

ActiveRecord::Base.connected_to(role: :writing) do
  # Your code here
end

A good example where this can be useful is if you are managing sessions in a database. If you do that, make sure you use the writing role. For example, you can override the save_record method in your session with authlogic:

class UserSession < Authlogic::Session::Base
  # Your session configuration

  def save_record(alternate_record = nil)
    ActiveRecord::Base.connected_to(role: :writing) do
      super
    end
  end
end

Another use case for automatic role-switching is when an app exposes a GraphQL API. All operations on a GraphQL API are POST, but conventionally, all queries are read-only, and mutations are read-write. In this case, to allow Rails to select the correct database, we can use a custom tracer in the schema.

First, create a tracer that switches roles based on the GraphQL document:

module Tracers
  class DatabaseRoleTracer
    EVENT_NAME = "execute_multiplex"

    # Execute on read only database if the operation has only queries.
    def trace(event, data)
      return yield unless Rails.env.production?
      return yield unless event == EVENT_NAME

      multiplex = data[:multiplex]
      ActiveRecord::Base.connected_to(role: role(multiplex)) do
        yield
      end
    end

    private

    def role(multiplex)
      if multiplex.queries.all?(&:query?)
        ActiveRecord::Base.reading_role
      else
        ActiveRecord::Base.writing_role
      end
    end
  end
end

And then use the tracer in your GraphQL Schema:

class MyAppSchema < GraphQL::Schema
  tracer Tracers::DatabaseRoleTracer.new

  # Your code here
end

Let’s get to the final alternative What can you do if your application still has a huge load on your primary database? The answer: database sharding.

Database Sharding in Your Rails Application

This is the most complex of all the methods for scaling databases discussed in this post. So reach for it only if you need it. It will add considerable complexity to your application and needs to be done right to provide any real benefit.

There are two strategies to shard your database:

Distribute different tables on different nodes (vertical sharding).
Have the same schema across all nodes, but distribute data depending on certain parameters (horizontal sharding).

Vertical sharding is similar to our “multiple databases” strategy in terms of setup and pros and cons, so we will not discuss it again.

Let’s discuss horizontal sharding and where it could be helpful.

One example is when you have a SaaS platform where a lot of data is associated with each user, but there is no overlap between the data of two users. In this case, splitting the data across nodes based on the user’s id is the most logical way to go. Every signed-in user will have to access only a single database, so we won’t have to reach out to multiple databases at the same time.

Setting up Horizontal Sharding

Let’s start with the configuration inside database.yml again:

production:
  primary:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_URL"] %>
  primary_replica:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_REPLICA_URL"] %>
  primary_shard_one:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_SHARD_ONE_URL"] %>
	primary_shard_one_replica:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_SHARD_ONE_REPLICA_URL"] %>

Then modify the ApplicationRecord to connect to different shards:

class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true

  connects_to shards: {
    default: { writing: :primary, reading: :primary_replica },
    shard_one: { writing: :primary_shard_one, reading: :primary_shard_one_replica }
  }
end

Now we can use the shard option with ActiveRecord::Base.connected_to to switch shards. Like the automatic database role selector, Rails also provides an automatic shard switcher that can be activated inside production.rb like this:

Rails.application.configure do
  config.active_record.shard_selector = { lock: true }
  config.active_record.shard_resolver = ->(request) { Tenant.find_by!(host: request.host).shard }
end

The shard_resolver lambda is the most interesting part. The above implementation relies on the assumption that our application is accessed from different domains/subdomains and distinguishes between the shards. Modify it to fit your application needs (you might need to access cookies to identify a user before switching the shard).

As with vertical scaling, this strategy is usually a one-way street. Once you shard a database, it is very hard to “un-shard” it (since several databases could have the same ids for different objects). But this lays down the foundation to really scale your app to millions or billions of users when fitting everything on the same server is not an option.

If you want to read more about database sharding, see this write-up by Linode.

Wrap Up

In this post, we discussed three major strategies to scale your database servers horizontally:

Splitting schema between multiple databases
Using read-only replica databases
Splitting data between multiple databases

All strategies have their pros and cons depending on your application use case and scale. Rails has really upped its database game in its last few releases. This post shows how easy it is to set up a Rails app to use multiple databases. As always, if your application allows it, I suggest starting small with optimizations. The easiest optimization to start with is a read-only replica, then only move on when you need more scale.

Happy scaling!

------------------

This article was originally posted on AppSignal Blog

State Machines in Ruby: An Introduction

Mon, 11 Jul 2022 05:28:33 GMT

A state machine can hold all possible states of something and the allowed transitions between these states. For example, the state machine for a door would have only two states (open and closed) and only two transitions (opening and closing).

On the other hand, complex state machines can have several different states with hundreds of transitions between them. In fact, if you look around, you will notice finite state machines surrounding you - when you buy from a website, get a pack of crisps from the vending machine, or even just withdraw money from an ATM.

In this post, we’ll look at how to set up a state machine in Ruby and use the state machines gem.

State Machines in Development

When do we need a state machine in development? The simple answer is whenever you want to model multiple rules for state transitions or perform side-effects on some transitions. The key here is to identify parts of your application that would benefit from a state machine. A good example that always works for me is an Order in the context of an e-commerce application.

For a simple application selling products online, an Order can be in one of several states:

created
processing
ready
shipped
delivered
void

You can see the allowed transitions in the following diagram.

Visualizing the order in the form of a state machine immediately allows us to clearly understand of the full flow, from ordering the product to delivery. And for us developers, it will enable clear, set steps on what can and cannot be done at particular points in that flow.

That being said, it is easy to jump down a rabbit hole by picking this up for a very wide problem and then end up with hundreds of states that are difficult to reason about and follow through. So always have a top-level idea and a state chart for your proposed state machine before implementing it.

Our First State Machine in Ruby

Let’s try to implement our OrderStateMachine with Ruby.

class OrderStateMachine
  def initialize(order)
    @order = order
    @order.state = :created if @order.state.blank?
  end

  def mark_as_paid!
    raise "Invalid state #{@order.state}" unless @order.created?

    @order.state = :processing
  end

  def packed!
    raise "Invalid state #{@order.state}" unless @order.processing?

    @order.state = :ready
  end

  def shipped!
    raise "Invalid state #{@order.state}" unless @order.ready?

    @order.state = :shipped
  end

  # ...
end

That’s a simple enough implementation. For each possible transition in the state machine, we define a new method that does some sanity checks and performs the transition.

The great thing about the above implementation is that everything is explicit: any new developer can very quickly understand the full extent of the state machine.

Let’s see how we can add some side-effects to the transitions. We’ll automatically ship orders that are packed and deliverable:

class OrderStateMachine
  # ...

  def packed!
    raise "Invalid state #{@order.state}" unless @order.processing?

    @order.state = :ready
    ship_order if @order.deliverable?
  end

  # ...

  private

  def ship_order
    DeliveryService.create_consigment!(@order)
    shipped!
  end
end

While a naïve implementation works great, it is overly verbose, especially when we have a lot of transitions and conditions.

A quick check on Ruby Toolbox brings up several gems for state machines. state_machines and aasm are the most popular ones, and both come with an ActiveRecord adapter if you want to use them with Rails. Both are thoroughly tested and production-ready, so do check them out if you need to implement a state machine.

Using the State Machines Gem in Ruby

For this post, I will describe how we can model the state machine for our Order using the state_machines gem.

class Order
  state_machine :state, initial: :created do
    event :confirm_payment do
      transition created: :processing
    end
    event :pack do
      transition processing: :ready
    end
    event :cancel do
      transition %i[created processing ready] => :void
    end
    event :return do
      transition delivered: :void
    end
    event :ship do
      transition ready: :shipped
    end
    event :fail_delivery do
      transition shipped: :processing
    end
  end
end

The above class defines our simple state machine and all of its possible transitions. On top of this, it also automatically exposes a lot of utility methods on an order:

order.state           # => "created"
order.state_name      # => :created
order.created?        # => true
order.can_pack?       # => false (since payment has not been confirmed yet)
order.confirm_payment # => true (and transitions to `processing`)
order.can_pack?       # => true

Execute Side-Effects with the State Machines Gem in Ruby

As is usual for any real-world system, many transitions in a state machine will come with some side effects. The state_machines gem makes it easy to define and execute them. Let’s add two side-effects:

On all transitions to ready, create a delivery consignment if the order is deliverable.
On all fail_delivery events, send an email to the user notifying them of the event.

class Order
  attr_accessor :deliverable

  state_machine :state, initial: :created do
    # ...

    after_transition to: :ready, do: :create_delivery_consignment, if: :deliverable
    after_transition on: :fail_delivery, do: :notify_delivery_failure
  end

  private

  def create_delivery_consignment
    DeliveryService.create_consigment!(self)
    ship
  end

  def notify_delivery_failure
    UserMailer.delivery_failure_email(self).deliver_later
  end
end

We can perform transitions as usual in the order:

order.deliverable = true
order.pack            # => true
order.state           # => "shipped" (through side-effect)

I would suggest checking out the Github page for the state_machines gem to learn about all the possible options that the gem offers.

Use the State Machines Gem with ActiveRecord in Ruby on Rails

As discussed before, both state_machines and aasm support ActiveRecord. When you use the state_machines gem with ActiveRecord, not a lot of things change with the implementation. You can continue using most of what we’ve already discussed in the previous sections of this post.

Here are some additional features that you get:

Side-effects happen inside a transaction. This means that if you do some database operations inside transition hooks and the transaction fails, the state change won’t be committed. See use_transactions if you want to disable this behavior.

Like all other ActiveRecord callbacks, if you want to update the attributes of the current model from a transition, you must do this inside a before_transition callback. To update attributes inside after_transition, you need to save the model again. Check out the following example:

class Order < ActiveRecord::Base
  state_machine initial: :created do

    before_transition any => :processing do |order|
      # You can just update the attribute here and it will be saved.
      order.processing_start_at = Time.zone.now
    end

    after_transition any => :shipped do |order|
      # Call update! to update the order.
      # Just setting the attribute like above would not work here.
      order.update!(shipped_at: Time.zone.now)
    end
  end
end

The gem automatically provides scopes to filter models by their state. For example, you can do Order.with_state(:processing) to find all the processing orders or Order.without_state(:processing) to find all orders that are not under processing.
If a state change is attempted without a matching transition (for example, from processing to delivered), the record will fail to save, and an error will be added to the validation errors. To internationalize the generated error messages, just add some keys to provide translations for states and events inside the config files (for example, en.yml):
```
en:
  activerecord:
    state_machines:
      order:
        states:
          created: Created
          processing: Under Processing
          # ...
        events:
          return: Return Order
          # ...
```

Wrap Up: Build State Machines in Ruby

In this post, we explored why we’d use a state machine in development, before building a simple state machine. Finally, we looked at using the state machines gem in Ruby and Ruby on Rails.

Now that you have a basic understanding of a state machine, I am sure you will recognize several scenarios around you that already use a state machine or will benefit greatly from one.

Until next time, I wish you luck building state machines!

This article was originally posted on AppSignal Blog

Using Profiling in Elixir to Improve Performance

Wed, 11 May 2022 08:58:28 GMT

Elixir is all about performance. Say you have an app up and running with Elixir, but some parts aren’t working as fast as you would like them to.

That is where profiling comes in. Profiling tools usually walk you through the frequency and duration of function calls and where they spend their time. Erlang has impressive profile tooling available at its disposal.

In this post, we will look into three profiling tools in Elixir: cprof, eprof, and fprof.

Do I Need Profiling for My Elixir App?

Firstly, I want to clarify that you may not need profiling at all. As Donald Knuth wrote: ‘premature optimization is the root of all evil.’ Some operations just cannot be optimized.

For example, say you want to fetch something from your database. You issue a query and get a reply. This incurs a network round trip with some data. If you need all the data, the network round trip time cannot be optimized without moving your database closer to the app.

With that out of the way, let’s jump into profiling options.

Profiling with `cprof` in Elixir

cprof counts the number of times each function is invoked. It doesn’t profile the time spent in those functions, so it comes with the least amount of overhead.

Use it when you already have an estimate of your functions’ runtimes. You’ll see the frequency with which each function is invoked.

Let’s see how we can profile a Fibonacci number generator with cprof.

You can follow along with the code. It uses the Rosetta Code algorithm for Fibonacci sequence generation.

Jump into an IEx session and run cprof like this:

iex(15)> :cprof.start()
19244
iex(16)> FibonacciRosettaCode.list(10)
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
iex(17)> :cprof.analyse(FibonacciRosettaCode)
{FibonacciRosettaCode, 79,
 [
   {{FibonacciRosettaCode, :fibonacci, 3}, 54},
   {{FibonacciRosettaCode, :fibonacci, 1}, 11},
   {{FibonacciRosettaCode, :"-fun.fibonacci/1-", 1}, 11},
   {{FibonacciRosettaCode, :module_info, 1}, 1},
   {{FibonacciRosettaCode, :list, 1}, 1},
   {{FibonacciRosettaCode, :__info__, 1}, 1}
 ]}
iex(18)> :cprof.stop()
19266

First start cprof, run the code that you want to profile, and then call :cprof.analyse/1 to fetch the statistics for the given module. Several other options are available inside the :cprof module, like pausing profiling with :cprof.pause/0.

We can also use Mix’s wrapper task profile.cprof directly.

➜ mix profile.cprof -e "FibonacciRosettaCode.list(10)"
                                                                     CNT
Total                                                                113
FibonacciRosettaCode                                                  77  <--
  FibonacciRosettaCode.fibonacci/3                                    54
  FibonacciRosettaCode.fibonacci/1                                    11
  FibonacciRosettaCode."-fun.fibonacci/1-"/1                          11
  FibonacciRosettaCode.list/1                                          1
Enum                                                                  24  <--
  Enum.reduce_range/5                                                 12
  anonymous fn/3 in Enum.map/2                                        11
  Enum.map/2                                                           1
  ...

Here, we can see that the function fibonacci/3 is called the most times. So if we can somehow decrease the number of calls to that function or optimize the code inside it, we could improve performance.

`eprof` Profiling in Elixir

cprof measures counts while eprof measures the execution time (in addition to counts) spent inside each function. It has slightly more overhead than cprof. Use eprof when you want to find the most time-consuming functions.

The usage is very similar to cprof. You just need to start the profiler, run the code you want to profile and then call analyze to fetch the results.

Let’s analyze the results from the mix task profile.eprof on our sample Fibonacci generator.

➜ mix profile.eprof -e "FibonacciRosettaCode.list(10)"
#                                               CALLS     % TIME µS/CALL
Total                                             106 100.0   45    0.42
anonymous fn/0 in :elixir_compiler_1.__FILE__/1     1  0.00    0    0.00
:lists.reverse/1                                    1  2.22    1    1.00
:lists.reverse/2                                    1  2.22    1    1.00
FibonacciRosettaCode.fibonacci/1                   11  4.44    2    0.18
Enum.map/2                                          1  4.44    2    2.00
Range.new/2                                         1  4.44    2    2.00
FibonacciRosettaCode."-fun.fibonacci/1-"/1         11  8.89    4    0.36
FibonacciRosettaCode.list/1                         1  8.89    4    4.00
:erlang.apply/2                                     1  8.89    4    4.00
anonymous fn/3 in Enum.map/2                       11 13.33    6    0.55
Enum.reduce_range/5                                12 13.33    6    0.50
FibonacciRosettaCode.fibonacci/3                   54 28.89   13    0.24

Here we can see that fibonacci/3 is again the most time-consuming part of our program, consuming 28.89% of the total execution time and 0.24µS per call.

In addition, eprof also allows us to profile function calls across different processes. The usage is very simple. Let’s see an example:

:eprof.start_profiling([self()])

# Do some work
1..100 |> Enum.each(fn i ->
  spawn(fn -> FibonacciRosettaCode.list(i + 1) end)
end)

:eprof.stop_profiling()
:eprof.analyze()

You need to call start_profiling/1 with a list of processes to profile. By default, this also tracks any other processes started from a profiled process.

When you are done, call stop_profiling and run analyze to get the results. They should look something like this, with an entry for each process and the percentage of time it was busy:

****** Process <0.150.0>    -- 37.26 % of profiled time ***
FUNCTION                                                             CALLS        %   TIME  [uS / CALLS]
--------                                                             -----  -------   ----  [----------]
io:request/2                                                             3     0.00      0  [      0.00]
...

****** Process <0.277.0>    -- 0.05 % of profiled time ***
FUNCTION                                             CALLS        %  TIME  [uS / CALLS]
--------                                             -----  -------  ----  [----------]
'Elixir.FibonacciRosettaCode':fibonacci/3                2     0.00     0  [      0.00]
'Elixir.FibonacciRosettaCode':fibonacci/1                3     3.03     1  [      0.33]
'Elixir.FibonacciRosettaCode':'-fun.fibonacci/1-'/1      3     3.03     1  [      0.33]
...

The great thing about this is that you can run it in production systems by accessing the remote console. Just call start_profiling/1, wait some time to process the requests or manually trigger some requests you want to profile. Then call stop_profiling followed by analyze to get the results.

Using `fprof` in Elixir

The final inbuilt tool you can use to profile your applications is fprof. It is a comprehensive profiling tool that generates a trace file containing timestamped entries for function calls, process-related events, and garbage collection data.

You can then feed this trace file into other tools to visualize the results thoroughly or use :fprof.analyse like above to fetch function counts and execution times.

Running fprof through IEx is quite advanced compared to running cprof and eprof, but fprof allows tracing multiple processes.

Here is how you can generate a trace file for all processes:

:fprof.start()
:fprof.trace([:start, procs: :all])

# Do some work
1..100 |> Enum.each(fn i ->
  spawn(fn -> FibonacciRosettaCode.list(i + 1) end)
end)

:fprof.trace(:stop)
:fprof.profile()
:fprof.analyse(totals: false, dest: 'prof.analysis')

This generates a comprehensive trace of everything that went on during the run with:

CNT - the number of times a function is called
ACC - the accumulated time spent in the function, including other function calls
OWN - the time the function spent to be executed.

Here’s the full file (50 MB) if you are interested.

Like eprof, fprof is a great tool to profile applications in production directly. Note that this will significantly slow down the application, so be prepared for degraded performance during profiling and a huge trace file.

So you have the data now. If you are feeling adventurous, you can start digging through the file and looking for patterns.

Or you can use some tools to help visualize your data. A popular one is erlgrind. It converts the fprof file to cgrind format that you can open inside KCachegrind.

To convert the file, download erlgrind and then run src/erlgrind profile.fprof. This will generate a prof.cgrind file that you can open to see graphs like this:

Profile and Collect Metrics from Your Elixir App in Production

eprof and fprof can assist you with profiling your application in production. But there are a couple of additional tools worth mentioning - Phoenix Live Dashboard and Recon.

Phoenix Live Dashboard

Phoenix’s live dashboard can provide a great, quick overview of metrics. While not exactly a profiler, it shows OS data and metrics from telemetry events and processes, among other things.

Here is an example of the dashboard in action:

Recon

Recon is another great library that can be dropped into any project and help diagnose problems in production. It works with very little overhead and provides production-safe tracing helpers.

You can simply install it by adding the recon dependency to mix.exs. You can then start tracing calls to any function by using :recon_trace.calls/3 like this:

:recon_trace.calls({FibonacciRosettaCode, :fibonacci, :return_trace}, 10)

This will trace all calls made to FibonacciRosettaCode.fibonacci for up to 10 lines, with one line for function call arguments and one for the return value with timestamps.

There is a lot more to recon, though, so I suggest you check out the Recon homepage for advanced usage.

AppSignal

AppSignal is another great tool to collect performance data (among other things). [Adding AppSignal to an existing application (https://docs.appsignal.com/elixir/installation/#installing-the-package) takes a few seconds. Just install the appsignal dependency and run the appsignal.install mix task with an API key. It has a good set of default metric collection including the throughput and response times for the application. You could even set up minutely probes to track custom metrics. Here’s an example of the AppSignal dashboard collecting data from a sample application:

If you are looking to collect data about specific blocks of code that you suspect are slow, AppSignal’s instrumentation feature can help collect data about it. Just wrap the suspected code inside Appsignal.instrument/2 like this:

defmodule MyApp.PageController do
  use MyApp, :controller

  def index(conn, _params) do
    custom_function()
    render(conn, "index.html")
  end

  defp custom_function do
    Appsignal.instrument("custom_function", fn ->
      :timer.sleep(1000)
    end)
  end
end

This will show up on the AppSignal event graph tagged as custom_function. You can then explore this event across multiple calls and make an informed decision about the need for optimization.

Check out the full list of features on the AppSignal docs.

Wrap-up: Measure Your Elixir App’s Performance with Profiling

In this post, we saw how inbuilt tools like cprof, eprof and fprof can help gather performance insights for your code, even during production.

We also took a quick look at a couple of other tools to monitor and trace your applications: Phoenix Live Dashboard and Recon.

Until next time, happy profiling!

This article was originally posted on AppSignal Blog

An Introduction to Pattern Matching in Ruby

Wed, 16 Feb 2022 05:57:32 GMT

Image from AppSignal Blog

Let’s start with a brief discussion about pattern matching in Ruby, what it does, and how it can help improve code readability.

If you are anything like me a few years ago, you might confuse it with pattern matching in Regex.

Even a quick Google search of ‘pattern matching’ without any other context brings you content that’s pretty close to that definition.

Formally, pattern matching is the process of checking any data (be it a sequence of characters, a series of tokens, a tuple, or anything else) against other data.

In terms of programming, depending on the capabilities of the language, this could mean any of the following:

Matching against an expected data type
Matching against an expected hash structure (e.g. presence of specific keys)
Matching against an expected array length
Assigning the matches (or a part of them) to some variables

My first foray into pattern matching was through Elixir. Elixir has first class support for pattern matching, so much so that the = operator is in fact the match operator, rather than simple assignment.

This means that in Elixir, the following is actually valid code:

iex> x = 1
iex> 1 = x

With that in mind, let’s look at the new pattern matching support for Ruby 2.7+ and how we can use it to make our code more readable, starting from today.

Ruby Pattern Matching with `case`/`in`

Ruby supports pattern matching with a special case/in expression. The syntax is:

case <expression>
in <pattern1>
  # ...
in <pattern2>
  # ...
else
  # ...
end

This is not to be confused with the case/when expression. when and in branches cannot be mixed in a single case.

If you do not provide an else expression, any failing match will raise a NoMatchingPatternError.

Pattern Matching Arrays in Ruby

Pattern matching can be used to match arrays to pre-required structures against data types, lengths or values.

For example, all of the following are matches (note that only the first in will be evaluated, as case stops looking after the first match):

case [1, 2, "Three"]
in [Integer, Integer, String]
  "matches"
in [1, 2, "Three"]
  "matches"
in [Integer, *]
  "matches" # because * is a spread operator that matches anything
in [a, *]
  "matches" # and the value of the variable a is now 1
end

A pattern matching clause like this is very useful when you want to produce multiple signals from a method call.

In the Elixir world, this is frequently used when performing operations that could have both an :ok result and an :error result, for example, inserted into a database.

Here is how we can use it for better readability:

def create
  case save(model_params)
  in [:ok, model]
    render :json => model
  in [:error, errors]
    render :json => errors
  end
end

# Somewhere in your code, e.g. inside a global helper or your model base class (with a different name).
def save(attrs)
  model = Model.new(attrs)
  model.save ? [:ok, model] : [:error, model.errors]
end

Pattern Matching Objects in Ruby

You can also match objects in Ruby to enforce a specific structure:

case {a: 1, b: 2}
in {a: Integer}
  "matches" # By default, all object matches are partial
in {a: Integer, **}
  "matches" # and is same as {a: Integer}
in {a: a}
  "matches" # and the value of variable a is now 1
in {a: Integer => a}
  "matches" # and the value of variable a is now 1
in {a: 1, b: b}
  "matches" # and the value of variable b is now 2
in {a: Integer, **nil}
  "does not match" # This will match only if the object has a and no other keys
end

This works great when imposing strong rules for matching against any params.

For example, if you are writing a fancy greeter, it could have the following (strongly opinionated) structure:

def greet(hash = {})
  case hash
  in {greeting: greeting, first_name: first_name, last_name: last_name}
    greet(greeting: greeting, name: "#{first_name} #{last_name}")
  in {greeting: greeting, name: name}
    puts "#{greeting}, #{name}"
  in {name: name}
    greet(greeting: "Hello", name: name)
  in {greeting: greeting}
    greet(greeting: greeting, name: "Anonymous")
  else
    greet(greeting: "Hello", name: "Anonymous")
  end
end

greet # Hello, Anonymous
greet(name: "John") # Hello, John
greet(first_name: "John", last_name: "Doe") # Hello, John Doe
greet(greeting: "Bonjour", first_name: "John", last_name: "Doe") # Bonjour, John Doe
greet(greeting: "Bonjour") # Bonjour, Anonymous

Variable Binding and Pinning in Ruby

As we have seen in some of the above examples, pattern matching is really useful to assign part of the patterns to arbitrary variables. This is called variable binding, and there are several ways we can bind to a variable:

With a strong type match, e.g. in [Integer => a] or in {a: Integer => a}
Without the type specification, e.g. in [a, 1, 2] or in {a: a}.
Without the variable name, which defaults to using the key name, e.g. in {a:} will define a variable named a with the value at key a.
Bind rest, e.g. in [Integer, *rest] or in {a: Integer, **rest}.

How, then, can we match when we want to use an existing variable as a sub-pattern? This is when we can use variable pinning with the ^ (pin) operator:

a = 1
case {a: 1, b: 2}
in {a: ^a}
  "matches"
end

You can even use this when a variable is defined in a pattern itself, allowing you to write powerful patterns like this:

case order
in {billing_address: {city:}, shipping_address: {city: ^city}}
  puts "both billing and shipping are to the same city"
else
  raise "both billing and shipping must be to the same city"
end

One important quirk to mention with variable binding is that even if the pattern doesn’t fully match, the variable will still have been bound. This can sometimes be useful.

But, in most cases, this could also be a cause of subtle bugs - so make sure that you don’t rely on shadowed variable values that have been used inside a match. For example, in the following, you would expect the city to be “Amsterdam”, but it would instead be “Berlin”:

city = "Amsterdam"
order = {billing_address: {city: "Berlin"}, shipping_address: {city: "Zurich"}}
case order
in {billing_address: {city:}, shipping_address: {city: ^city}}
  puts "both billing and shipping are to the same city"
else
  puts "both billing and shipping must be to the same city"
end
puts city # Berlin instead of Amsterdam

Matching Ruby’s Custom Classes

You can implement some special methods to make custom classes pattern matching aware in Ruby.

For example, to pattern match a user against his first_name and last_name, we can define deconstruct_keys on the class:

class User
  def deconstruct_keys(keys)
    {first_name: first_name, last_name: last_name}
  end
end

case user
in {first_name: "John"}
  puts "Hey, John"
end

The keys argument to deconstruct_keys contains the keys that have been requested in the pattern. This is a way for the receiver to provide only the required keys if computing all of them is expensive.

In the same way as deconstruct_keys, we could provide an implementation of deconstruct to allow objects to be pattern matched as an array. For example, let’s say we have a Location class that has latitude and longitude. In addition to using deconstruct_keys to provide latitude and longitude keys, we could expose an array in the form of [latitude, longitude] as well:

class Location
  def deconstruct
    [latitude, longitude]
  end
end

case location
in [Float => latitude, Float => longitude]
  puts "#{latitude}, #{longitude}"
end

Using Guards for Complex Patterns

If we have complex patterns that cannot be represented with regular pattern match operators, we can also use an if (or unless) statement to provide a guard for the match:

case [1, 2]
in [a, b] if b == a * 2
  "matches"
else
  "no match"
end

Pattern Matching with `=>`/`in` Without `case`

If you are on Ruby 3+, you have access to even more pattern matching magic. Starting from Ruby 3, pattern matching can be done in a single line without a case statement:

[1, 2, "Three"] => [Integer => one, two, String => three]
puts one # 1
puts two # 2
puts three # Three

# Same as above
[1, 2, "Three"] in [Integer => one, two, String => three]

Given that the above syntax does not have an else clause, it is most useful when the data structure is known beforehand.

As an example, this pattern could fit well inside a base controller that allows only admin users:

class AdminController < AuthenticatedController
  before_action :verify_admin

  private

  def verify_admin
    Current.user => {role: :admin}
  rescue NoMatchingPatternError
    raise NotAllowedError
  end
end

Pattern Matching in Ruby: Watch This Space

At first, pattern matching can feel a bit strange to grasp. To some, it might feel like glorified object/array deconstruction.

But if the popularity of Elixir is any indication, pattern matching is a great tool to have in your arsenal. And having first-hand experience with using it on Elixir, I can confirm that it is hard to live without once you get used to it.

If you are on Ruby 2.7, pattern matching (with case/in) is still experimental. With Ruby 3, case/in has moved to stable while the newly introduced single-line pattern matching expressions are experimental. Warnings can be turned off with Warning[:experimental] = false in code or -W:no-experimental command-line key.

Even though pattern matching in Ruby is still in its early stages, I hope you’ve found this introduction useful and that you’re as excited as I am about future developments to come!

This article was originally posted on AppSignal Blog

What's new in Rails 7

Wed, 15 Dec 2021 10:57:37 GMT

Rails 7 is just around the corner. We don’t have a confirmed release date, but it is expected to be available before Christmas, so not very long to go. The latest version as of this post’s publication is 7.0.0.rc1, the first release candidate. Basecamp, HEY, Github, and Shopify have all been running the Rails 7 alpha in production, so we can expect even the release candidate to be pretty stable.

Rails 7 was released on December 15.

In this post, we will look at some of the new features and changes that Rails 7 will bring.

Node and Webpack Not Required

Yes, you read that right! JavaScript in Rails 7 will no longer require NodeJS or Webpack. And you can still use npm packages.

Transpiling ES6 with Babel and bundling with Webpack require a lot of setup. While Rails supported it pretty well with the Webpacker gem, this brings a lot of baggage, is hard to understand and make any changes to, especially while maintaining upgradability.

The default for new apps created with rails new is now to use import maps through the importmaps-rails gem. Instead of writing a package.json and installing dependencies with npm or yarn, you use ./bin/importmap CLI to pin (or unpin or update) dependencies.

For example, to install date-fns:

$ ./bin/importmap pin date-fns

This will add a line in config/importmap.rb like:

pin "date-fns", to: "https://ga.jspm.io/npm:date-fns@2.27.0/esm/index.js"

In your JavaScript code, you can continue using everything like you used to:

import { formatDistance, subDays } from 'date-fns'

formatDistance(subDays(new Date(), 3), new Date(), { addSuffix: true })
//=> "3 days ago"

One thing to keep in mind with this setup is that there is no transpiling between what you write and what the browser gets. For the most part, this is ok since all browsers that matter now support ES6 out of the box.

But this also means that you won’t be able to use TypeScript or jsx because they require transpilation to JS before use.

So, if you want to use React with JSX, you still have to fall back to a different setup (using webpack/rollup/esbuild).

Rails 7 can do this for you. All you need is one command with your chosen strategy:

$ ./bin/rails javascript:install:[esbuild|rollup|webpack]

Turbolinks and UJS Replaced by Turbo and Stimulus

Applications generated with Rails 7 will get Turbo and Stimulus (from Hotwire) by default, instead of Turbolinks and UJS. Hotwire is a new approach that delivers fast updates to the DOM by sending HTML over the wire.

Encryption at Database Layer

Rails 7 allows marking certain database fields as encrypted using the encrypts method on ActiveRecord::Base. This means that after an initial setup, you can write code like this:

class Message < ApplicationRecord
  encrypts :text
end

You can continue using the encrypted attributes like you would any other attribute. Rails 7 will encrypt and decrypt it automatically between the database and your application.

But this comes with a slight quirk: you cannot query the database by that field unless you pass a deterministic: true option to the encrypts method. The deterministic mode is less secure than the default non-deterministic mode, so only use it for attributes you absolutely need to query.

Asynchronous Querying

There is now a load_async method that you can use when querying data to fetch results in the background. This is especially important when you need to load several un-related queries from a controller action. You can run:

def PostsController
  def index
    @posts = Post.load_async
    @categories = Category.load_async
  end
end

This will fire both queries in the background at the same time. So, if each query takes 200ms, the total time spent fetching the data is ~200ms instead of 400ms, if they are fetched serially.

Zeitwerk Mode for Rails 7

This is a breaking change for older applications that still run the classic loader. All Rails 7 applications must use Zeitwerk mode, but the switch is pretty easy. Check out the full Zeitwerk upgrade guide.

Other Rails 7 Updates

Retry Jobs Unlimited Times

ActiveJob now allows passing :unlimited as the attempts parameter on retry_on. Rails will continue to attempt the job without any maximum number of attempts.

class MyJob < ActiveJob::Base
  retry_on(AlwaysRetryException, attempts: :unlimited)

  def perform
    raise "KABOOM"
  end
end

Named Variants

You can now name variants on ActiveStorage instead of specifying size on every access.

class User < ApplicationRecord
  has_one_attached :avatar do |attachable|
    attachable.variant :thumb, resize: "100x100"
  end
end

#Call avatar.variant(:thumb) to get a thumb variant of an avatar:
<%= image_tag user.avatar.variant(:thumb) %>

Hash to HTML Attributes

There is a new tag.attributes method for use in views that translates a hash into HTML attributes:

<input <%= tag.attributes(type: :text, aria: { label: "Search" }) %>>

will produce

<input type="text" aria-label="Search">

Ruby `debug`

The new default for debugging has changed from byebug to the debug gem.

Instead of calling byebug, you now need to call debugger in the code to enter a debugging session.

Assert a Single Record with `sole`

When querying records, you can now call sole or find_sole_by (instead of first or find_by) when you want to assert that the query should only match a single record.

Product.where(["price = %?", price]).sole
# => ActiveRecord::RecordNotFound      (if no Product with given price)
# => #<Product ...>                    (if one Product with given price)
# => ActiveRecord::SoleRecordExceeded  (if more than one Product with given price)

user.api_keys.find_sole_by(key: key)
# as above

Check Presence / Absence of an Association

We can now use where.associated(:association) to check if an association is present on a record instead of joining and checking for the existence of an id.

# Before:
account.users.joins(:contact).where.not(contact_id: nil)

# After:
account.users.where.associated(:contact)

Stream Generated Files from Controller Actions

You can now use send_stream inside a controller action to start streaming a file that is being generated on the fly.

send_stream(filename: "subscribers.csv") do |stream|
  stream.write "email_address,updated_at\n"

  @subscribers.find_each do |subscriber|
    stream.write "#{subscriber.email_address},#{subscriber.updated_at}\n"
  end
end

This provides an immediate (partial) response to the user so that they know something is happening and has an added benefit if you deploy on Heroku.

Since the file will start streaming immediately, Heroku will not terminate the connection. This means you don’t need to resort to background jobs to generate one-off files that take longer than 30 seconds.

Upgrading to Rails 7

As with previous versions of Rails, upgrading is simple. While we don’t have an official upgrade guide yet, the steps will remain the same:

Change the Rails version number in the Gemfile (7.0.0.rc1 as of the publication date) and run bundle update.
Run bundle exec rails app:update. Follow the interactive CLI and add/replace/modify the files as required.
Run your tests and verify everything works as expected.

Wrap up

You can see the full list of bug fixes, features, and changes in the Rails 7 release notes. These are not comprehensive at the moment, but we can expect them to be updated soon.

If you are still running Rails 6 or lower, please note that with the final release of Rails 7, Rails 6.1 will enter the “security issues only” mode and will no longer receive bug fixes. This will also mark the EOL for Rails 5.2, as it will no longer receive any fixes.

Have fun coding!

This article was originally posted on AppSignal Blog

Testing WebSocket Clients in Elixir with a Mock Server

Sun, 01 Aug 2021 06:14:58 GMT

Image from AppSignal Blog

Introduction

In this post we will discuss about a very high level overview of implementing a long running connection between two services with the use of WebSocket and then writing unit tests for the functionality.

The WebSocket protocol allows a two-way communication between two channels. Most developers would know it as a fast, near real-time communication medium between the client and the server. In the Phoenix and Elixir world, many would recognize it from LiveView or Channels which are abstractions built on top of fast realtime transports, usually a WebSocket.

But another area that WebSocket is a great communication medium for is an app built on the Microservices architecture. Let’s say we have an e-commerce application that uses (among others) two services:

Orders - Create and manage the user’s orders
Payments - Processing the payments.

When a user makes a purchase, the Orders service would create an order corresponding to the purchase and send it to the Payments service. This service could then respond with a payment page URL which the user then needs to visit to complete his payment. After the completion, the Orders service needs to create an invoice for the order and send it to the user, so it will need a way to watch for the completion of payments.

For such an architecture, a long running connection between Orders and Payments through a WebSocket is a great choice as both services can interchange messages and notify the other service about updates or events.

WebSocket Client

WebSockex is a library to implement a WebSocket client in Elixir. A very simplified implementation of the client as it could look on the Orders service can be found here.

The client behaves as follows:

It is started as a process with a URL of the Payments WebSocket Server and some details about the order it needs to process.
As soon as it is connected, it sends a message to the Payments service to initiate the payment process.
At some point (normally soon after the payment has been initialized), the Payments service responds with a list of available payment methods for the order.
The client selects the first payment method and asks Payments to initiate payment with that method.
Payments sends a message with the payment page url that the Orders service can use to initiate a payment on the web app.
At some point, Payments sends a message to the client informing about the state of the transaction, e.g. FULFILL, CANCEL etc.
If the payment is FULFILL, fulfill the order. If the payment is CANCEL, cancel the order and close the connection.

Testing External Services

Now that we have a working WebSocket client in place, the next question is how do we test it? It connects to an external service (even though this service is ultimately controlled by us, but this is still external in terms of the Orders service). There are several solutions for writing tests that interact with external services:

Create a separate service for the API Client and Stub out those requests with Mox from José Valim.
Mock out the requests during the tests with a library like Mock.
Roll out your own web server to handle the requests.

All these methods and their pros and cons have been covered in detail in several posts in the community. One that I particularly like and recommend is Testing External Web Requests in Elixir?.

We will discuss how to write tests for our above client by rolling out our own server during the tests.

Mock Server

To test out the client, we first need to create a Mock Server that will be able to provide a connection to this client and interact with it. We will use plug and cowboy to create the server and control the socket.

Mock HTTP Server

First, since we might need many of these servers running in parallel during the tests, we will need a way to generate ports so that the servers don’t collide. This can be done by starting an Agent that starts randomly at a port and then assigns us a port incrementally during the tests

defp get_port do
  unless Process.whereis(__MODULE__), do: start_ports_agent()

  Agent.get_and_update(__MODULE__, fn port -> {port, port + 1} end)
end

defp start_ports_agent do
  Agent.start(fn -> Enum.random(50_000..63_000) end, name: __MODULE__)
end

The Mock Server now needs to obtain a port, start listening on that port for connections. On each new client connection, we will open a socket (which we will see later). This is the server side socket that is linked with the client side socket being opened from our WebSocket client implementation. This socket is running in its own process, so in order for our test code to interact with it, we will need to obtain its PID inside the tests. Since the tests only know about the HTTP Server and not the socket, we will send the Socket PID back to the server so we can retrieve it from the test code. Here is how the Mock Server code looks (I know it is a bit complex, so don’t be alarmed, it is easy to understand once we see the socket).

The usage of the above server is very simple:

Start the server:

{:ok, {server_ref, url}} = Commerce.Orders.MockWebsocketServer.start(self())
on_exit(fn -> Commerce.Orders.MockWebsocketServer.shutdown(server_ref) end)

Connect the client using the url from above.

If you need to send messages to the client, first obtain the server pid

server_pid = Commerce.Orders.MockWebsocketServer.receive_socket_pid()

Send Messages to the server that will relay them to the client. Check Commerce.Orders.TestSocket.websocket_info/2 for all possible clauses
```
send(server_pid, {:send, {:text, frame}})
```
This server also sends all messages that it receives to the owner process. This means that to verify that the server has received a message, you can use
```
assert_receive on the owner:
assert_receive("SOME MESSAGE")
```

Mock Socket

The next thing after creating the mock server is to create the actual server side socket that will handle the connections. This is the Commerce.Orders.TestSocket to which our mock HTTP server dispatches the initial connection to.

The full code for the test socket can be found here. Let’s break this down.

The init is what is called after the MockWebsocketServer is started and it receives a new connection request from our WebSocket client (that we want to test). You can check out the full cowboy_websocket documentation for more details about the supported responses. [{test_pid, agent_pid}] is the initial state that we sent out from the MockWebsocketServer.dispatch and we will keep track of that in state to manage sending of frames to the socket.

websocket_init is called once the connection is upgraded to WebSocket and this is where we send own (i.e. the Server Socket’s) pid to the MockWebsocketServer which the test code can then retrieve using receive_socket_pid.

websocket_handle is called for every frame received. This is where we further process the received frame to verify messages from our client. If this differs between different WebSocket clients that we want to test, you could skip the call to handle_websocket_message and it’s implementations and just use send(state.pid, to_string(msg)) to send all messages to the test process. The test process can then verify the frames as required.

websocket_info is called for every Erlang message that the process receives. This is what we will be using to receive messages from the test code to trigger certain events from the Server Socket. The first argument to this function could be anything that the test code sends, so you can add as many clauses as you would like to support to simulate certain events. Here we are interested in only initiating a close from the test code or sending a particular frame to the client socket.

Test Code

Now that we have the WebSocket client and the Mock Server and Mock Socket out of the way, we can get to the actual test code that will test the WebSocket client using the Mocks.

To start the server in our tests, we will use MockWebsocketServer.start:

setup do
  {:ok, {server_ref, url}} = MockWebsocketServer.start(self())
  on_exit(fn -> MockWebsocketServer.shutdown(server_ref) end)

  %{url: url, server_ref: server_ref}
end

Let’s test out our client with the mock server in place. Instead of several smaller tests that test a single part of the client, I am writing a big test case that describes the full interaction for the sake of describing all available testing options without too much boilerplate code.

test "interacts with the server", %{url: url, order: order} do
  {:ok, pid} = WebSocketClient.start_link(%{url: url})
  server_pid = MockWebsocketServer.receive_socket_pid()

  # A payment is initiated immediately after a connection
  assert_receive Jason.encode!(%{initiate_payment: true})

  # WebSocket client receives the payment methods from the server (from the hard coded message response). We do not need any code for this since we generalized it directly in the socket.

  # The payment is initiated with credit card
  assert_receive Jason.encode!(%{initiate_payment: "credit_card"})

  # Send a FULFILL state from the server. This is the alternative to hard coded messages in the server side socket.
  send(server_pid, {:send, {:text, Jason.encode!(%{state: "FULFILL"})}})

  # Confirm that the order was fulfilled
  assert {:ok, %Order{state: "FULFILL"}} = Orders.get_order(order.id)
end

Wrap-up

In this post we touched briefly on a sample Microservices architecture, how we could use WebSockets to communicate between different services and finally saw a very simplified implementation of a WebSocket client using WebSockex. If you are looking for a more complex WebSockex client example, I recommend this post which walks through building a complete STOMP client with WebSocket.

While it took some work to set up our mock server and socket for testing our WebSocket clients, it makes our testing really smooth and simple. If we add another WebSocket client later, we could utilize the same server to unit-test the new client without any complex setup or mocking.

In addition to this, we now have tests that do not depend on any implementation details of the client. We are just testing business logic rather than implementation or tying our test code to specific libraries being used in the implementation. So if you were to replace the WebSocket client implementation to use gun instead of WebSockex, we wouldn’t need to change anything in our tests. In fact, our tests would serve as an additional validation point for this migration to see that everything keeps working like it did before.

This article was originally posted on AppSignal Blog

Monitoring Application Email Delivery

Sun, 02 May 2021 06:14:58 GMT

Email Delivery Services

If you have ever used Mailgun or another similar email delivery service like SendGrid etc with a shared IP pool, you would have noticed that some emails are occasionally blocked even though your own spam rate is very low or zero. This usually happens because someone else assigned on the same shared IP as yours has been abusing the services by sending spam emails and the IP has been blacklisted. While Mailgun and SendGrid do actively trace and resolve these issues, but this still means that you are missing emails to your users, and if all you are sending are transactional emails that have high value to the users, this will affect their confidence in the service.

The easiest way to tackle these issues is to opt for a dedicated IP from the service providers. You would then be in control of your IP reputation and can avoid such issues in the future if you are sending good emails.

But a dedicated IP does not make sense for all teams. You might be starting a new project or have a very limited amounts of emails to deliver even on an already established product. In this case, it might be better to roll out your own email monitoring system, especially since it is very easy to manage and maintain.

Email Monitoring

The main use case of our monitoring solution is to track email failures and retry the email (which might obtain a new IP from the shared pool and thereby avoid the blacklisting). In case we are experiencing repeated failures, it might be a good idea to be notified so that we can manually intervene.

While all of this can be done in any framework, since I had recently the need for it in an Elixir/Phoenix app, that is what I will describe. I will be assuming Bamboo as our email library but this should again work even if you are using Swoosh or another library.

Email Schema

The first step is to create a schema to track all emails that you are queueing from the application. Let’s call it MyApp.Communication.Email. Since this step will probably be different for a lot of apps, I will just give a very brief list of the fields I have considered for the schema.

schema "emails" do
  field :attempt_at, :utc_datetime
  field :attempt_count, :integer
  field :mailgun_id, :string
  field :mailgun_url, :string
  field :delivered_recipients, {:array, :string}

  embeds_one :last_delivery_status, DeliveryStatus, on_replace: :delete do
    field :code, :integer
    field :message, :string
    field :attempt_no, :integer
    field :description, :string
  end

  timestamps()
end

Feel free to add or remove the fields as per your needs.

Tracking Queued Emails

The next step is to configure the Mailer to add your own method that will track the email after it has been queued to Mailgun.

@doc """
Delivers an email immediately and tracks its delivery.
"""
def deliver_now_with_tracking!(email) do
  case deliver_now(email, response: true) do
    {:error, error} ->
      raise error
    {:ok, _email, response} ->
      track_email(response)
      :ok
  end
end

@doc """
Queues an email for delivery and tracks its delivery.
"""
def deliver_later_with_tracking(nil), do: {:error, "no email"}
def deliver_later_with_tracking(email) do
  if Mix.env() == :test do
      deliver_now_with_tracking!(email)
  else
    Task.Supervisor.start_child(Bamboo.TaskSupervisor, fn ->
      deliver_now_with_tracking!(email)
    end)
  end
end

I assume this is pretty self explanatory. In addition to Bamboo’s deliver_now! and deliver_later, we now have two more methods that allow to track the email. The only thing of note here is we are passing response: true to Bamboo’s deliver methods to obtain the raw response from Mailgun. Mailgun responds with a json in the format {"id": "<message_id>"}. Here is our tracking code that just creates a new Email entry in the database:

defp track_email({:ok, %{"id" => id}}), do: MyApp.Communication.create_email(%{
  mailgun_id: id |> String.trim_leading("<") |> String.trim_trailing(">"),
  attempt_count: 1,
  attempt_at: Timex.now()
})
defp track_email(%{body: body, status_code: 200}), do: track_email(Jason.decode(body))
defp track_email(%Bamboo.Email{}) do
  if Mix.env() == :test, do: track_email(%{body: "{\"id\": \"<ID@mx.domain.ch>\"}", status_code: 200})
end
defp track_email(response), do: Logger.debug("Failed to track email. Response not recognized - #{inspect(response)}")

Receiving Mailgun Notifications

The next step is to create a controller that receives notifications from Mailgun through Webhook. I will leave out the router configuration since that will vary from app to app. Just keep in mind that you will need to use a pipeline that DOES NOT contain protect_from_forgery as the requests are originating from Mailgun which will not have that info.

Mailgun notifications are formatted like this:

{
  "signature": {"token": "xx", "timestamp": 123, "signature": "xx"},
  "event-data": {
    "event": "delivered|failed",
    "message": {"headers": {"message-id": "xx"}},
    "recipient": "xx",
    "delivery-status": {},
    "storage": {"url": "xx"}
  }
}

Inside the controller, there are two main aspects to responding to Mailgun notifications.

Verify the signature: Here is how you can compute the signature on your app. You need to verify this with the signature that Mailgun sends in the json payload .

defp compute_signature(token, timestamp),
    do: :sha256 |> :crypto.hmac(mailgun_webhook_signing_key(), token <> timestamp) |> Base.encode16(case: :lower)

Handle the event: This is where things get interesting. We are only going to track the delivered and failed events as these are the most interesting ones for us.

defp handle_event(%{"event" => "delivered", "recipient" => recipient}, %Email{} = email), do: Communication.update_email(email, %{recipient: recipient})
defp handle_event(%{"event" => "failed", "delivery-status" => status, "storage" => %{"url" => url}, "recipient" => recipient} = event, %Email{} = email) do
  if retryable?(status, email) do
    attrs = %{
      last_delivery_status: status,
      attempt_at: Timex.now() |> Timex.shift(minutes: 4 |> :math.pow(email.attempt_count) |> floor()),
      mailgun_url: url,
      to: recipient
    }
    Communication.retry_email(email, attrs)
  else
    title = MyAppWeb.Gettext.dgettext("emails", "Email Delivery Abandoned")
    MyAppWeb.AdminEmail.email_alert(title, title, event) |> MyAppWeb.Mailer.deliver_later()
    {:ok, email}
  end
end
defp handle_event(_event, _email), do: {:error, :not_acceptable}

Here, if the email is delivered, we just add the recipient to our tracked email. If the email is undelivered, we do a check if this is one of the emails that we want to retry (a simple logic could be to retry all emails with status between 500 and 554 and less than 5 attempts.

If it is, we retry it with an exponential duration or discard it and send an alert email to the admin so that he can manually intervene and check on it.

Now, to our final step. How do we retry the email. Mailgun makes it really simple. Each email has a unique storage URL (this is the mailgun_url that we tracked on our email model). We can post to that URL with some form data to re-enqueue the email with new details. Here is a simple method on the mailer that does this:

@doc """
Schedules an email on Mailgun. Requires:
  * url - Mailgun Storage URL (like https://storage.eu.mailgun.net/v3/domains/mg.domain.ch/messages/BASE64KEY)
  * datetime - UTC Date Time to schedule
  * to - Recipient Address
Returns the raw response from Mailgun.

## Examples:
    iex> schedule_mailgun_email([api_key: "foo"], %{url: "xyz", datetime: ~U[], to: "some@example.com"})
    %{body: "{\"id\": \"<ID@mx.domain.ch>\"}"}
"""
def schedule_mailgun_email!(config, %{url: url, datetime: datetime, to: to}) do
  auth_key = Base.encode64("api:" <> config.api_key)
  HTTPoison.post!(
    url,
    {:form, [{"to", to}, {"o:deliverytime", datetime |> Timex.to_unix()}]},
    [Authorization: "Basic #{auth_key}"],
    [ssl: [{:versions, [:'tlsv1.2']}], follow_redirect: true]
  )
end

With this system in place, you are ready to configure Mailgun to send webhook notifications (this can be configured from the Dashboard -> Sending -> Webhook) to your app and you would be ready to automatically track and retry any failed emails.

Elixir Pooling with Poolboy

Sun, 18 Apr 2021 09:42:58 GMT

Poolboy is a generic pooling library for Erlang (ergo Elixir). Since it is originally made for Erlang, you wouldn’t find a great set of documentation around how to use it on Elixir apps, and especially with Phoenix apps. If you know your way around Erlang, you can follow the wonderful documentation on their readme. If not, this post will try to explain just the basics of how you could get started and include this in your project.

First things first, do you need pooling? If you are spinning up processes that could take a while to run, you could soon run into a problem where there is a huge wave of processes being spinned up without the previous ones finishing and thereby exhausting all your system resources. BEAM is great at distributing work across the processes and it will try to do its best, but even it is limited by system resources. This is where Poolboy comes in.

The installation is simple, something like {:poolboy, "~> 1.5"} in your mix.exs would normally be enough. Now, in order to use Poolboy, you would first need to identify the resource that you need to pool. If you have a static module that you spin up to perform some task, e.g. an open connection to a database (all of the following is imaginary code), something like the following:

defmodule MyApp.Processor do
  def process(arg1, arg2) do
		{:ok, connection} = DBConnection.open()
		DBConnection.do_something(connection, arg1, arg2)
	end
end

There are two big issues with the above code.

You are opening a new connection to the DB whenever you want to process something.
If you have 1000s of requests at the same time, you might soon run out of memory if your DB does not allow that many simultaneous connections.

This is of course an imaginary situation and most DB libraries out there would probably manage this for you. But this could be extended to any number of use cases, for example, opening a ErlPort to a ruby process and communicating with it.

Let’s see how we can pool this operation. First, we will need to make the module stateful. The easiest way to do that is to create an Agent. Let’s see how the agent would look like:

defmodule MyApp.ProcessorAgent do
	def start_link(_params) do
    Agent.start_link(fn ->
      {:ok, connection} = DBConnection.open()
      connection
    end)
  end

	def process(pid, arg1, arg2) do
		Agent.get(pid, fn connection ->
      DBConnection.do_something(connection, arg1, arg2)
    end)
	end
end

Simple enough, right? We create a new state in start_link and then whenever we need to process something, we get the agent from it’s PID and do something with the connection (that is the state of the agent). We can now update our original Processor to use this agent instead of directly opening a new connection.

defmodule MyApp.Processor do
  def process(arg1, arg2) do
		:poolboy.transaction(:processor_agent, fn pid ->
      MyApp.ProcessorAgent.process(pid, arg1, arg2)
    end)
	end
end

This uses the :poolboy.transaction/3 to use a checkout a process from a pool and run the processing logic inside that. You might be wondering what is special about the :processor_agent atom. Nothing, this is just a name that we give to the process pool when starting our application. In application.ex, do the following inside your start/2 method:

def start(_type, _args) do
	children = [
		# ...
		:poolboy.child_spec(:processor_agent, processor_poolboy_config())]
	]
end

def processor_poolboy_config() do
	[
		name: {:local, :processor_agent},
		worker_module: MyApp.ProcessorAgent,
		size: 5, # Initial number of workers
		max_overflow: 2 # Extra workers (auto-stopped after work) to spawn if under load
	]
end

There are a few other options to configure Poolboy. Now that you know how the interface with Elixir should look like, I will leave you to explore the configuration options on Poolboy.

Hosting NextJS projects on Heroku with SSL

Sat, 13 Mar 2021 18:00:00 GMT

I am part of a team where we are building a NextJS web app and not long ago, we were looking for deployment options. Since most of our infrastructure is on Heroku, this was the obvious first choice for us since the team had had good experience with it in the fast and we wanted to focus on keeping the dev-ops to a minimum. The main requirements for our app were that we were going to be using wildcard domains and we absolutely needed SSL for the app.

If there was a single domain, Heroku is pretty straightforward as SSL and automatic SSL redirects work out of the box. For our use-case however, even though Heroku does provide wildcard SSL certificates with their automated SSL management, it was prohibitively expensive to obtain one. So we wanted to use an external certificate for the website. Setting up external certificates is pretty easy on Heroku, but unfortunately, it doesn’t provide automatic SSL redirects. So if a user lands on the http page, you either have to resort to client side redirects (for this, by the way, react-https-redirect is one of the easiest to set up) or fiddle with the buildpack configuration to enable automatic redirects.

We decided to go with the second approach, by using the nginx-buildpack on top of the heroku/nodejs buildpack that we used to build the NextJS app. First step, add the buildpack to your app:

$ heroku buildpacks:add heroku-community/nginx

For what we wanted to achieve, we would need the “Customizable NGINX config” feature from the buildpack. Here is how our config/nginx.config.erb looks like:

daemon off;
# Heroku dynos have at least 4 cores.
worker_processes <%= ENV['NGINX_WORKERS'] || 4 %>;

events {
	use epoll;
	accept_mutex on;
	worker_connections <%= ENV['NGINX_WORKER_CONNECTIONS'] || 1024 %>;
}

http {
	gzip on;
	gzip_comp_level 2;
	gzip_min_length 512;

	server_tokens off;

	log_format l2met 'measure#nginx.service=$request_time request_id=$http_x_request_id';
	access_log <%= ENV['NGINX_ACCESS_LOG_PATH'] || 'logs/nginx/access.log' %> l2met;
	error_log <%= ENV['NGINX_ERROR_LOG_PATH'] || 'logs/nginx/error.log' %>;

	include mime.types;
	default_type application/octet-stream;
	sendfile on;

	# Must read the body in 5 seconds.
	client_body_timeout 5;

	server {
		listen <%= ENV["PORT"] %>;
		server_name _;
		keepalive_timeout 5;

		location / {

      <% if ENV['NGINX_SKIP_HTTPS_PROXY'] == 'true' %>
        if ($http_x_forwarded_proto != "https") {
          return 301 https://$host$request_uri;
        }
      <% end %>

			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header Host $http_host;
			proxy_redirect off;
			proxy_pass http://localhost:3000; #next serve listens here and receives nginx requests
		}
	}
}

All other portions of this config are pretty straightforward. The main part where the magic happens is this:

location / {

  <% if ENV['NGINX_SKIP_HTTPS_PROXY'] == 'true' %>
    if ($http_x_forwarded_proto != "https") {
      return 301 https://$host$request_uri;
    }
  <% end %>

  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_redirect off;
  proxy_pass http://localhost:3000; #next serve listens here and receives nginx requests
}

This catches all paths accessed on the http protocol, we redirect with a 301 to the corresponding path on https. This is where things get interesting. Even with the https protocol, Heroku always forwards the request to the http block since under the hood, Heroku router (over)writes the X-Forwarded-Proto and the X-Forwarded-Port request headers. So, if the header value of X-Forwarded-Proto, we should not redirect and instead set up a reverse-proxy to our deployed app.

Conditional global stylesheets for Next.js apps

Sun, 01 Nov 2020 18:00:00 GMT

Next.js gives you the best developer experience with all the features you need for production: hybrid static & server rendering, TypeScript support, smart bundling, route pre-fetching, and more. No config needed. The out of the box CSS support is amazing. It means that you could just put your css/scss files that you obtained from the designer at styles/scss/app.scss (the path or the file name don’t matter) and add the following into your _app.tsx (or your _app.jsx).

import '../styles/scss/app.scss';

This sets everything up including the compilation of SCSS files and hot reloading of the styles whenever you make some changes.

The Problem

But the shortcoming with this approach is that this import needs to be inside the global _app file. You cannot import it inside individual pages if you need to separate styles by page (or have a different stylesheet for some of the pages in your app). A common use case for this feature is if your app has two components, an admin interface where special users could perform some operations on your domain objects and a public interface which is visible to everyone else. Of course, you could ask the designer to provide a global stylesheet that works for both parts of the interface, but that is not always possible (and unnecessarily wasteful).

Styled JSX (and Babel and Webpack Configuration)

This is where Next.js’s styled-jsx support comes in. Bear in mind, it isn’t exactly meant to solve this issue, but it works great nonetheless. First start by installing the required packages (not that you don’t need sass related plugins if you have a CSS stylesheet):

yarn add styled-jsx @types/styled-jsx styled-jsx-plugin-sass node-sass-middleware sass

Next you need to configure babel for styled-jsx (again, you don’t need the sass specific stuff if you have only the CSS). Make sure your .babelrc looks something like this:

{
  "presets": [
    [
      "next/babel",
      {
        "styled-jsx": {
          "plugins": [
            [ "styled-jsx-plugin-sass", { "sassOptions": { "includePaths": ["./styles"] } }]
          ]
        }
      }
    ]
  ]
}

Now configure webpack (the config lives inside next.config.js) to allow loading css and scss files with the styled-jsx/webpack loader.

module.exports = {
  webpack(config, { defaultLoaders }) {
    config.module.rules.push({
      test: /\.(scss|css)$/,
      use: [
        defaultLoaders.babel,
        {
          loader: require('styled-jsx/webpack').loader,
          options: {
            type: 'global',
          },
        },
      ],
    });

    return config;
  },
};

The Style Component

Now we are ready to use the stylesheets inside our app. All we need is import the styles from the respective files and provide them through a regular react component.

import publicStyles from '../../../styles/public.scss';
import adminStyles from '../../../styles/admin.scss';

function AdminStyles(): React.ReactElement {
  return <style jsx global>{adminStyles}</style>;
}

function PublicStyles(): React.ReactElement {
  return <style jsx global>{publicStyles}</style>;
}

export default function AppStyles({ admin }: { admin: boolean }): React.ReactElement {
  if (admin) { return <AdminStyles />; }
  return <PublicStyles />;
}

You can now include this component as a part of your App exported from _app.tsx.

function App({ Component, pageProps }: Props): React.ReactElement {
  const router = useRouter();
  const isAdmin = router.pathname.startsWith('/admin');

  return (
    <>
      <Component {...pageProps} />
      <AppStyles admin={isAdmin} />
    </>
  );
}

Authentication Using GraphQL Ruby

Mon, 12 Oct 2020 18:00:00 GMT

I often get asked questions about how to implement an authentication system for use in a frontend app relying wholly on GraphQL. There are divided opinions on whether you should keep authentication separate from the GraphQL API. To be honest, I don’t see a reason to do that. If you do authentication through a controller reached through a REST API or through the GraphQL Schema, it hardly makes any difference and if the frontend relies heavily on GraphQL, I don’t see a reason not to have the authentication part with it too.

Separating GraphQL Schema

The first thing to handling authentication through GraphQL is serve two different Schemas based on whether the user is making authenticated requests or a non-authenticated one. It is also the simplest one to take care of, so let’s get this out of the way.

If you are using graphql-ruby, you might already have something along the lines of the following in your GraphQLController:

def execute_query
  schema.execute(:document => document,
                  :variables => @variables,
                  :context => @context,
                  :operation_name => @operation_name)
        .to_json
end

All we need is to provide that schema class based on whether the user is currently logged in. Something like this:

def schema
  @schema ||= if Current.user
                Schema::AuthenticatedSchema
              else
                Schema::PublicSchema
              end
end

Just define the schema classes as separate classes inheriting from GraphQL::Schema and you are done.

Public Schema and Auth Mutations

The first question we need to ask ourselves is what authentication is in terms of our app? For us, it is a mutation that creates a new UserSession on the backend. So the PublicSchema boils down to a single mutation:

module Schema
  class PublicSchema < GraphQL::Schema
    mutation Types::PublicMutationType
  end
end

module Types
  class PublicMutationType < BaseObject
    description "Public Mutation API"

    field :login, :mutation => Mutations::Login
  end
end

You can add more operations if you need to support sign up, account recovery options etc. But once we can figure out the login part, the rest could be derived pretty easily.

module Mutations
  class Login < Base
    graphql_name "Login"

    argument :email, String, :required => false
    argument :password, String, :required => false
    argument :remember_token, String, :required => false

    field :session, UserSessionType, :null => true
    field :errors, [String], :null => true

    def resolve(args)
      if args[:remember_token].present?
        user = User.find_by(:persistence_token => args[:remember_token])
        user_session = UserSession.new(user) if user.present?
      else
        user_session = UserSession.new(args.slice(:email, :password))
      end

      return { :errors => ["unauthorized"] } unless user_session.save
      return unless user_session.user

      { :session => user_session, :name => user.name }
    end
  end
end

Here’s how it can be implemented easily. Note that we are using authlogic as our authentication system. If you are using something else, only the resolve implementation would change.

Unit Testing Hooks containing GraphQL Queries

Sun, 20 Sep 2020 18:00:00 GMT

If you jumped on the hooks train on react, you would know that it is exciting. But React officially doesn’t (yet) have a way to test hooks. Yes, you could set up a dummy component that renders the hook and test it out. But soon, you want to do some advance handling inside the hook that the dummy component doesn’t allow testing for.

Testing Hooks

Before we jump on to testing hooks containing GraphQL queries, let’s get this out of the way. What if you want to test just a simple hook and you are seeing the following error:

Invariant Violation: Hooks can only be called inside the body of a function component.

The react-hooks-testing-library provides a renderHook method that allows to render the hook in a stand alone component for testing. The usage is very simple, just return the results of the hook from the callback of renderHook. The current return value of the hook will be present in result.current. So if your hook returns a number (count) and a method (increment) to update that count, you could write:

test('should increment counter', () => {
  const { result } = renderHook(() => useCounter())
  expect(result.current.count).toBe(0);

  act(() => {
    result.current.increment()
  })
  expect(result.current.count).toBe(1)
});

Hooks with GraphQL Queries

So we know it is simple to test regular hooks. What if there’s a hook that contains some useQuery statements, or uses client.query from an effect. Something along the lines of:

export function useModelWithNestedFetch({ id, query }) {
  const client = useApolloClient();
  const { data, ...queryState } = useQuery(query, { variables: { id, skip: id === 'new' });
  const [model, setModel] = useState(id === 'new' ? newModel() : data?.model);

  useEffect(() => {
    setModel(id === 'new' ? newModel() : data?.model);

    const nestedIds = data?.model?.nestedIds;
    if (!nestedIds?.length) { return data?.model && setModel(update(data.model, { nesteds: { $set: [] } })); }

    // Load nesteds
    client.query({
      query: gqlDynamic`
        query Nesteds {
          ${nestedIds.map((id) => `
          nested_${id}: nested(id:${id}) {
            ...NestedInfo
          }
          `).join('\n')}
        }
      `,
    })
      .then((nestedData) => setModel(update(data.model, { nesteds: { $set: Object.values(nestedData.data) } })))
      .catch((error) => console.warn('failed to fetch nesteds', error));
  }, [id, data, client, setModel]);
  return { model, setModel, ...queryState };
}

I know this is a lot of code, but we never said we were testing a simple hook, did we. Let’s first see what it is doing. We are first fetching (or initializing if it is a new record) a model. The API, unfortunately, doesn’t embed the nested records and returns only nestedIds inside the field. So after fetching the record, we run another query that is generated at runtime to fetch each of the nested records and then use setModel callback to update or fetched model with the loaded nested records.

Now comes the interesting part, how do we test this?

The first thing that we need to do is mock out the queries. But it is difficult to do that inside a hook. So what we will do instead is mock out those hooks themselves. It is easy with jest.mock:

jest.mock('@apollo/react-hooks', () => {
  const { nested1 } = require('../../../fixtures/nesteds');
  const { model1 } = require('../../../fixtures/models');

  const data = {
    model: model1,
  };
  const fetchMore = jest.fn();
  const client = {
    query: jest.fn(() => new Promise((resolve) => {
      setTimeout(() => resolve({ data: { nested1 } }), 100);
    })),
  };
  return {
    __esModule: true,
    useMutation: jest.fn(() => [jest.fn(), null]),
    useQuery: jest.fn(() => ({ data, fetchMore })),
    useApolloClient: jest.fn(() => client),
  };
});

Now, time for our first test, where we will just test that the new model is initialized properly

it('doesn\'t query and returns new model when id new', () => {
  const { result } = renderHook(() => useModelWithNestedFetch({ id : 'new', query: 'QUERY' }));
  expect(result.current.model).toEqual(newModel());
  expect(useQuery.mock.calls[0][1].skip).toBeTruthy();
});

The second test will validate that we query for our model and then after an effect, we also query for the nested records. Here we will use the waitForNextUpdate from the testing library’s renderHook return value to wait for the effect to update the state.

it('queries for model', async () => {
  const { result, waitForNextUpdate } = renderHook(() => useModelWithNestedFetch({ id : 1, query: 'QUERY' }));
  expect(result.current.model).toEqual(model1);
  expect(useQuery.mock.calls[0][1].skip).toBeFalsy();
  expect(useQuery.mock.calls[0][1].variables.id).toBe(1);

  expect(useApolloClient.mock.results[0].value.query).toHaveBeenCalledTimes(1);

  act(() => jest.advanceTimersByTime(100));
  await waitForNextUpdate();

  expect(result.current.model.nesteds).toEqual([nested1]);
});

If you are following along, you might be wondering why your tests are intermittently failing, right? That is because, we are missing a very important part of the setup. You remember we returned a promise from the Apollo client.query mock? That promise would sometimes resolve before the next statement and sometimes not. We need to use fake timers to make sure the promises/timeouts are properly handled:

beforeEach(() => {
  jest.useFakeTimers();
  [
    useMutation,
    useQuery,
    useApolloClient,
  ].forEach(mock => mock.mockClear());
});

Using Action Mailbox on Rails 6

Wed, 08 Jul 2020 18:00:00 GMT

Action Mailbox is a framework available with Rails 6 to handle incoming emails with Rails. It integrates with a variety of service providers out of the box including SendGrid, Mandrill, Mailgun, or self hosted Exim, Postfix or Qmail. It stores all emails in the database and also provides declarative routing for mails to custom Mailboxes. A great usecase we have for this feature is to allow imports (or file uploads, for example) to certain users directly through email.

The first step is to install action_mailbox to the application. Run rails action_mailbox:install. If everything went well, this will create some new migrations and a file app/mailboxes/application_mailbox.rb. Run the migrations to create the required tables on your DB.

Routing Emails

The routes for the emails are defined in application_mailbox. The rules are processed from top to bottom and the mail is delivered to the first matched mailbox. Here’s a sample route definition that forwards all emails coming to import-{key}@example.com to ImportMailbox and all others to a BounceMailbox.

class ApplicationMailbox < ActionMailbox::Base
  routing(/import-(\w+)@#{ENV.fetch("INBOUND_EMAIL_DOMAIN", "example.com")}\Z/i => :import)

  # bounce all other emails
  routing(:all => :bounce)
end

Create mailboxes

To handle our incoming emails, we will need to create custom mailboxes. We can have Rails generate them for us using rails g mailbox <<name>>.

From our routing through ApplicationMailbox, the emails coming with import-xyz@example.com will automatically call ImportMailbox#process. We can then access mail details using:

mail.from - Array
mail.recipients - Array
mail.subject - String
See body and attachments methods below for other details

class ImportMailbox < ApplicationMailbox
  def process
    # Handle mail here
  end

  def body
    mail.body.decoded
  end

  def attachments
    mail.attachments.map do |attachment|
      # access attachment.fiilename, attachment.decoded, attachment.content_type
      # You can also add the attachments to a model (e.g. `Resource` here)  connected to ActiveStorage
      Resource.new(:name => attachment.filename).tap do |r|
        r.attachment.attach(:io => StringIO.new(attachment.decoded),
                            :filename => attachment.filename,
                            :content_type => attachment.content_type)
        r.save!
      end
    end
  end
end

If you need to bounce emails (for example, if they don’t match any user’s address), you can use bounce_with with reference to an email (can be generated using ActionMailer):

bounce_with BounceMailer.invalid_to_address(mail.from.first)

Unit testing mailboxes

The official ActionMailbox docs are very light about the topic of unit testing mailboxes. ActionMailbox::TestHelper provides some utility methods to test emails. The core methods of importance are receive_inbound_email_from_fixture (which reads from an eml file fixture) and receive_inbound_email_from_mail which accepts a block to configure the email.

receive_inbound_email_from_mail do |mail|
  mail.to "To Name <to@example.com>"
  mail.from "From Name <from@bagend.com>"
  mail.subject "Test subject"
  mail.text_part do |part|
    part.body "test body"
  end
  mail.html_part do |part|
    part.body "<h1>test body</h1>"
  end
end

The core of the testing depends on the ruby mail library. So if you find that the test helper isn’t working for you or has some limitations (for example, I couldn’t get it to work with attachments), you can always fall back to the Mail implementation. Here is my sample implementation:

mail = Mail.new do
  from    "from@example.com"
  to      "to@example.com"
  cc      "cc@example.com"
  subject "Test subject"
  content_type  "multipart/mixed"
  part(:content_type => "multipart/alternative") do |p|
    p.part "text/html" do |p|
      p.body = "test body"
    end

    p.part "text/plain" do |p|
      p.body = "test body"
    end
  end
end

mail.attachments["attachment.csv"] = { :mime_type => "text/csv", :content => "a,b,c" }
inbound = ActionMailbox::InboundEmail.create_and_extract_message_id!(mail.to_s, :status => :processing)

inbound.route

Send notifications with Firebase from Firestore events using Cloud Functions

Sat, 20 Jun 2020 09:57:00 GMT

Firebase is great for quick MVPs when you don’t want to invest too much time on the server side. Let’s look at one of the most powerful Firebase features, Cloud Functions. With this, you can write any code that will be executed on Firebase’s servers to create custom web hooks or to respond to changes in the database.

Firestore Schema

Let’s take example of a chat app and see how we can send notifications to users on new incoming messages with these functions. It depends heavily on the database schema, so let’s quickly go over the schema we will be handling with Firestore.

So whenever a new message is sent, we create a new Thread and add it to the active channel.

Function

The first step is to set up your project for development. Firebase has a good guide on it.

We will use the onCreate function on firestore document to be notified whenever a new record is created in a collection.

exports.sendChatNotifications = functions.firestore.document('channels/{cid}/thread/{did}').onCreate(async (threadSnapshot, context) => {
  // ...
});

The next step is to find all the participants of the channel this message was sent to.

const participants = await admin.firestore().collection('channel_participation').where('channel', '==', context.params.cid).get();

After we find the participants, we will use sendToDevice method on the admin messaging API to actually send out the notification. Don’t forget the sound key to make sure the notification makes a sound on all devices. We can also include custom image and icon in the notification by passing urls to images. Here is the full code with everything put together.

const functions = require('firebase-functions');
const admin = require('firebase-admin');

admin.initializeApp({
  credential: admin.credential.applicationDefault(),
  databaseURL: '...'
});

exports.sendChatNotifications = functions.firestore.document('channels/{cid}/thread/{did}').onCreate(async (threadSnapshot, context) => {
  const thread = threadSnapshot.data();
  const senderId = thread.senderID;

  console.log(`Finding channel participants with channel ${context.params.cid}`);
  const participants = await admin.firestore().collection('channel_participation').where('channel', '==', context.params.cid).get();
  if (participants.empty) {
    console.log('There are no participants to send to.');
    return;
  }

  participants.forEach(async (snapshot) => {
    const uid = snapshot.data().user;
    if (uid === senderId) { return; }

    const user = await admin.firestore().doc(`users/${uid}`).get();

    const payload = {
      notification: {
        title: thread.senderFirstName,
        body: thread.content,
        icon: thread.senderProfilePictureURL,
        image: thread.url,
        sound: 'default',
      }
    };
    await admin.messaging().sendToDevice([user.data().fcmToken], payload);
  });
});

All we need now is to deploy this function. We can do that with yarn firebase deploy --only=functions

Unit Test custom React Hooks that rely on Promise

Sat, 06 Jun 2020 09:57:00 GMT

So you wrote a fancy custom hook that fetches some information from an API. How do we unit test such hooks?

Simple Hooks

Before we get to testing async hooks, you should already know how to test hooks. There are already some posts about it elsewhere (this one is an excellent example), so I will not repeat all that was already said there. We will use the react-hooks-testing-library for this purpose. Here’s an example from the library:

`useCounter.js`

import { useState, useCallback } from 'react'

function useCounter() {
  const [count, setCount] = useState(0)

  const increment = useCallback(() => setCount((x) => x + 1), [])

  return { count, increment }
}

export default useCounter

`useCounter.test.js`

import { renderHook, act } from '@testing-library/react-hooks'
import useCounter from './useCounter'

test('should increment counter', () => {
  const { result } = renderHook(() => useCounter())

  act(() => {
    result.current.increment()
  })

  expect(result.current.count).toBe(1)
})

Testing async hooks

Assume that instead of a local counter, we are fetching the count from a GraphQL API using Apollo Client’s query method. I know we could use Apollo’s react hook directly, but this is a contrived example as the async part could be anything.

`useCounter.js`

import { useQuery, useApolloClient } from "@apollo/react-hooks";
import { gql } from "graphql.macro";
import { useEffect } from "react";

const QUERY = gql`query Counter($id:ID!) { counter(id: $id) { count } }`;

export function useCounter(id) {
  const [count, setCount] = useState();
  const client = useApolloClient();
  useEffect(() => {
    client.query({
      query: QUERY,
      variables: { id }
    })
      .then(data => setCount(data.counter.count))
      .catch(error => console.log('failed to fetch count', error));
  }, [client, id]);
  return count;
}

In order to test this hook, we need a way to have a mocked result being passed to the query. First, we need to mock out the client and its query function. You need to change it to mock out your actual calls. I will mock out the query with a promise that resolves to some data after 100ms.

jest.mock('@apollo/react-hooks', () => {
  const client = {
    query: jest.fn(() => new Promise((resolve) => {
      setTimeout(() => resolve({ data: { counter: { count: 1 } } }), 100);
    })),
  };
  return {
    __esModule: true,
    useApolloClient: jest.fn(() => client),
  };
});

We created a promise that resolves after 100 ms. Now, we could wait for 100ms in our tests and then check the result after that time. But that doesn’t really work when we have a lot of tests or the promise could resolve before we get to the next statement if something takes time on the test framework. We will instead use fake timers. We will do this is a beforeEach to execute before each test in the file.

beforeEach(() => {
  jest.useFakeTimers();
  useApolloClient.mockClear();
});

Now to the actual test which is actually pretty self-explanatory.

it('fetches and resolves to a count', async () => {
  const { result, waitForNextUpdate } = renderHook(() => useCounter('1');
  expect(result.current).toBeFalsy();
  expect(useApolloClient.mock.results[0].value.query).toHaveBeenCalled();

  act(() => jest.advanceTimersByTime(100));
  await waitForNextUpdate();

  expect(result.current).toEqual(1);
});

Transform SVG icons and images to React Components

Tue, 12 May 2020 09:57:00 GMT

Introduction

There are plenty of ways to include SVG files in your React apps. One that I was using until recently was to include the SVG with an object tag. While it worked ok, there’s a much better way. What if you could include the SVG files directly in the generated markup?

Enter SVGR

SVGR provides a way to generate react components from SVG files that you can easily import in your code. It comes built-in with create-react-app, so if your code is generated from it, you don’t need to read further. In fact, the official docs cover the default behavior pretty well and it is easy to set up.

But if you are looking for a solution to generate tsx files from SVG, you are in the right place. From the documentation, svgr has an option to specify a template that you want it to use.

A custom template takes place in a file that exports a “template function”.

This function is called in a babel plugin: babel-plugin-transform-svg-component and must returns a Babel AST. If you are not familiar with all Babel stuff, you should read this guide.

Reading the Babel AST guide, we can prepare our template so that it writes out a typescript file instead of vanilla JS.

function defaultTemplate(
  { template },
  opts,
  { imports, interfaces, componentName, props, jsx, exports },
) {
  const plugins = ['jsx']
  if (opts.typescript) {
    plugins.push('typescript')
  }
  const typeScriptTpl = template.smart({ plugins })
  return typeScriptTpl.ast`${imports}
${interfaces}
function ${componentName}(${props}) {
  return ${jsx};
}
${exports}
  `
}
module.exports = defaultTemplate

Now use this template when processing the SVG files:

$ yarn svgr --template template.js play.svg

Here’s the output of play.tsx:

import * as React from "react";

function SvgPlay(props) {
  return (
    <svg baseProfile="tiny" viewBox="0 0 24 34" {...props}>
      <path d="M7 22.2V11.8c0-.8.9-1.2 1.5-.8l8.1 5.2c.6.4.6 1.3 0 1.7L8.5 23c-.6.4-1.5 0-1.5-.8z" />
    </svg>
  );
}

export default SvgPlay;

Automation

If you are looking to automate the generation of these files so that you don’t have to run these commands every time a new SVG file is added, here’s how you can do that with Grunt (it’s possible with many other tools, including gulp, but our project was using Grunt).

We need to use the grunt-shell task with this config.

svgr: {
  command: 'yarn svgr --ext tsx --template lib/misc/svgr_template.js -d lib/src/svg lib/svg',
}

Apollo Client Default Values for Missing Fields

Mon, 13 Apr 2020 04:58:27 GMT

GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.

But what if you want to ask for incremental data updates (e.g. as part of a subscription). There is no way in GraphQL to listen for incremental data from a subscription. Of course, you can ask for all data and then the server could be coded to transmit incremental updates. But this doesn’t work on the client-side if you are using Apollo Client because it expects all data that you have queried for in your subscription.

There are some libraries on the that try to provide a way to remove duplicates from GraphQL responses, e.g. graphql-deduplicator but it doesn’t have support for merging missing fields.

Here is a quick implementation that defaults all missing fields to a null value. You can then use your own implementation to merge the values as you see fit, but this gets rid of the Apollo Client errors of the kind Missing field X.

The first step is to add a method to your Model (it assumes that all your models inherit from a base Model) objects that you want to handle.

class Model {
  // Converts missing fields to nulls to play well with Apollo client
  static fillUndefineds(clazz: typeof Model, object: any): any {
    if (!object) { return object; }
    const meta = clazz.meta as any;
    _.forEach(meta.attributes, ({ id, clazz }: any, key) => {
      if (typeof object[id] === 'undefined') {
        object[id] = null;
      } else if (object[id] && clazz) {
        object[id] = Model.fillUndefineds(clazz, object[id]);
      }
    });
    return object;
  }
}

The next step is to mark some meta attributes on the models you are trying to convert. You can create your own format for the meta, but here is something that I have used in the past.

class Movie extends Model {}
Movie.meta = {
  attributes: {
    name: { id: 'name', type: TYPE_STRING }
  }
}

class Event extends Model {}
Event.meta = {
  attributes: {
    movie: { id: 'movie', clazz: Movie }
  }
}

Finally, we can create a custom SubscriptionClient and set it on our Link.

// To automatically convert missing backend values to null, use this client
// This doesn't suit well for partial objects (which contain only the changed data)
// but can be used if the app is sure that it will receive a full object.
class WSSubscriptionClient extends SubscriptionClient {
  request(req: any): Observable<any> {
    const parentObservable = super.request(req);
    const observable = {
      subscribe: (observer: Observer<any>) => parentObservable.subscribe({
        next: (value: any) => {
          if (!value || !value.data || !value.data.event) { return observer.next(value); }
          const event = Model.fillUndefineds(Event, value.data.event);
          return observer.next(_.extend({}, value, { data: { event } }));
        },
        error: (err) => observer.error && observer.error(err),
        complete: () => observer.complete && observer.complete(),
      })
    };
    return observable;
  }
}

const subscriptionClient = new WSSubscriptionClient(url, { reconnect: true });
const wsLink = new WebSocketLink(subscriptionClient);
const apolloClient = new ApolloClient({ link: wsLink, cache: new InMemoryCache() });

Advanced Mixins on Typescript

Thu, 28 Mar 2019 15:01:34 GMT

Mixins on Typescript

TypeScript 2.2 first introduced the support for mixins to huge applause from the developers. Mixins pattern allows you to abstract code into separate chunks and DRY code is always better, right? A very simple example of a mixin from the official TypeScript docs:

class Point {
    constructor(public x: number, public y: number) {}
}

class Person {
    constructor(public name: string) {}
}

type Constructor<T> = new(...args: any[]) => T;

function Tagged<T extends Constructor<{}>>(Base: T) {
    return class extends Base {
        _tag: string;
        constructor(...args: any[]) {
            super(...args);
            this._tag = "";
        }
    }
}

const TaggedPoint = Tagged(Point);

let point = new TaggedPoint(10, 20);
point._tag = "hello";

class Customer extends Tagged(Person) {
    accountBalance: number;
}

let customer = new Customer("Joe");
customer._tag = "test";
customer.accountBalance = 0;

So in short, mixins can be used to supercharge your classes with extra attributes and methods. But what if you want to inject a common functionality to several similar classes that all implement a particular interface. For example, let’s say you have a mixin that produces some sound.

// Activatable Mixin
class Speakable {
    speak() {
      // Speak something
    }
}

This can then be injected into a Dog and a Cat to make them speak. But what if the speak method needs to call a specific method sound on the instances? While this is possible with TypeScript, it is very hard to set up. Here is how an implementation for this would look like.

interface Animal {
  sound: string;
  speak: () => void;
}

type Constructor<T = Animal> = new (...args: any[]) => T;

export default function Speakable<TBase extends Constructor>(Base: TBase) {
  return class extends Base {
    speak(this.sound);
  };
}

class BaseDog extends Animal {
  sound = 'wow';
  speak() { throw new Error('Method not implemented.'); }
}

class Dog extends Speakable(BaseDog) {}

class BaseCat extends Animal {
  sound = 'meow';
  speak() { throw new Error('Method not implemented'); }
}

class Cat extends Speakable(BaseCat) {}

Let’s break this down. We have an interface Animal that is supposed to have a sound and speak something. But since the speak implementation is very involved and same for all Animals, we want to provide an abstraction for it rather than having to define it on all sub-classes of Animal. On the contrived example, you would say let’s create a helper function that both classes can call. While that sounds ok for a small example like this, when it gets to the point of providing several functionalities that depend on one another, say speak, walk, run, cry etc. it becomes really difficult to manage it with helper functions.

The mixins approach, on the other hand, helps us compose classes as we would while allowing us to provide default implementations for the common methods into a single mixin. We could have several different mixins that can be nested one after the other to provide different independent functionalities too.

class Cat extends Taggable(Speakable(BaseCat)) {}

While this does look slightly hard to read, the benefit in terms of DRYness far outweights the small issue of aesthetics.

Custom fetch implementation for responses compressed with MessagePack::RackMiddleware

Fri, 28 Dec 2018 13:49:42 GMT

After the previous post, I have received some messages on how to use the responses passed through the custom middleware. So if you are not one of the guys using that middleware, this post is not for you. If you are, you’re in luck, keep reading.

The easiest approach, if you don’t have a lot of requests going to the compressed API, as some have suggested is to manually add the header to the request and uncompress it when you receive a response. But this can soon get tiring for multiple requests and needs a better way to handle this, especially since it’s so easy.

We can create a custom fetch that makes sure that this custom header is included in the request and parse the response if it is compressed. It has the same API as the native fetch, so it’s easy to get used to.

Depending on what browsers you target, you might not need bluebird, request-promise and whatwg-fetch. This is from a code from a nodejs app which does require the latter two.

import 'whatwg-fetch';
import Promise = require('bluebird');
import _ = require('lodash');
import msgpack = require('msgpack-lite');
import request = require('request-promise');

function nativeFetchJson(url: string, options: any = null): Promise<any> {
  return fetch(url, options).then(response => response.json());
}

export default function fetchJson(url: string, options: any = null): Promise<any> {
  const compressionEnabled = _.get(options, 'compressionEnabled');
  delete options.compressionEnabled;
  if (!compressionEnabled) { return nativeFetchJson(url, options); }
  return request.get(url,
    _.defaultsDeep({}, options, {
      headers: { 'x-messagepack-compression': true },
      resolveWithFullResponse: true,
      encoding: null
    }),
  ).then((response) => {
    if (response.headers['x-messagepack-compression'] !== 'true') { return JSON.parse(response.body); }
    return msgpack.decode(response.body);
  });
}

declare function fetch(url: string): Promise<any>;
declare function fetch(url: string, options: any): Promise<any>;

Comment below if you want me to do a version without all of the dependencies and I’ll give it a shot.

Custom Rack Middleware for response compression

Tue, 27 Nov 2018 18:23:00 GMT

If you are serving a Rails application, you are probably already using some sort of compression for the responses. If not, enabling simple gzip compression is pretty straightforward (using a simple middleware) and can give boosts of as much as 80% in some cases. You can put this in an initializer somewhere and your responses will be gzipped if the receiver supports it.

config.middleware.insert_after ActionDispatch::Static, Rack::Deflater

The middleware automatically detects when compression is supported and allowed. For example, no transformation is made when a cache directive of ‘no-transform’ is present, or when the response status code is one that doesn’t allow an entity body.

But, if you have a really specific use case, you can go one step further. For example, if you deal mostly in JSON responses, messagepack could help your API get even leaner. A word of caution before you venture further, don’t expect huge savings if you are already using gzip compression as messagepack savings can vary hugely based on the kind of data you are transmitting over the socket.

Let’s create a custom middleware that can compress responses with messagepack before sending them out. A few things that we need to keep in mind for this extension are:

It only compresses if there’s a specific header field (X-MessagePack-Compression) in the request signaling that it wants a compressed response.
Since messagepack only works on JSON, make sure that it is executed only for JSON responses.

With these conditions, here’s a sample middleware that you can put in.

module MessagePack
  class RackMiddleware
    def initialize(app)
      @app = app
    end

    def call(env)
      status, headers, body = @app.call(env)
      return [status, headers, body] unless compress?(env, headers)

      suppress(Exception) do
        body = MessagePack.pack(Oj.load(body.body))
        headers["X-MessagePack-Compression"] = true
      end
      [status, headers, [body]]
    end

    private

    def compress?(env, headers)
      env["HTTP_X_MESSAGEPACK_COMPRESSION"] &&
        headers["Content-Type"].include?("application/json;")
    end
  end
end

Using the middleware is easy, you just need one extra line during initialization:

Rails.application.config.middleware.use MessagePack::RackMiddleware

Implement Magic Login for TV devices

Tue, 30 Oct 2018 11:04:49 GMT

Introduction

I don’t know if that is the correct term for it, but it sure is magic to non-developers. I recently noticed this technique on the Youtube app on my TV. Here’s how it works:

Begin the login process on Youtube app. It shows a unique 4 character code for the TV.
Open Google app on your phone and navigate to the “authenticate other devices” page.
It asks for a confirmation for the TV login on the mobile without entering anything and logs you in after the confirmation.

How

So how does it work? It’s pretty simple, really. As soon as you open the authentication screen on the TV, it generates a unique code and starts playing an ultrasonic sound in a loop. The phone is capable of hearing this sound and identifying what code it is broadcasting. After that, it’s all a matter of communicating the right things over the API on the TV and the phone.

Implementation

Let’s see how we can do this. We will use the Chirp SDK for this purpose. The SDK will do all the heavy lifting of generating and identifying the tones, so all we need is to build a way for it.

TV App

Let’s start with the implementation on the TV. Since most of the devices use a different programming language, I wouldn’t dive into a language-specific guide. But for completeness, here’s a sample 4 liner in JS:

const audioContext = new (window.AudioContext || window.webkitAudioContext)();
const connect = new ChirpConnect(APP_KEY, audioContext);

const payload = new Uint8Array([0, 1, 2, 3]) // TODO: Replace with your code
connect.send(payload)

If your device isn’t compatible in generating the payload, fret not, because there’s an API that does that too. All you need is to download the file that https://audio.chirp.io/v3/ultrasonic/<<YOUR HEX ENCODED CODE>> generates and play it on a loop on your TV app.

Mobile App

The next step is to integrate it into your mobile app so you can detect these tones and perform your magic code for logging in the user. Again, setting up the SDK is as simple as it gets. Do this somewhere in viewWillAppear:

connect = ChirpConnect(appKey: APP_KEY, andSecret: APP_SECRET)
let configError: Error = connect.setConfig(<YOUR CONFIG STRING>)
let startError: Error = connect.start()

connect.receivedBlock = {
  (data : Data?, channel: UInt?) -> () in
  guard let data = data else { return }
  print("Received code \(String(data: data, encoding: .ascii))")
  // TODO: Initiate Login
}

The receivedBlock will be executed with the data that it receives and you can initiate the login process from there.

Of course, all of this needs to be aided with APIs on the backend to generate a unique code and then log the user in if requested from an authenticated end-point, but those are details that I will leave for you as the backend for most apps will very widely.

The Ocean at the End of the Lane Review

Tue, 02 Oct 2018 15:11:52 GMT

Sussex, England. A middle-aged man returns to his childhood home to attend a funeral. Although the house he lived in is long gone, he is drawn to the farm at the end of the road, where, when he was seven, he encountered a most remarkable girl, Lettie Hempstock, and her mother and grandmother. He hasn’t thought of Lettie in decades, and yet as he sits by the pond (a pond that she’d claimed was an ocean) behind the ramshackle old farmhouse, the unremembered past comes flooding back. And it is a past too strange, too frightening, too dangerous to have happened to anyone, let alone a small boy.

As we age, we become our parents; live long enough and we see faces repeat in time.

If this is what childhood fantasy is like, I need more. Never knew this was a genre I could be interested in. A fantastic story, amazingly well written in first person voice.

At about 156 pages, it’s a short read. But no way I would have expected to finish it as quickly as I did. It’s a real page turner and a very difficult book to put down. I won’t dive into the story as I usually do for other reviews because I don’t think I can give it justice. Just take dive in this Ocean and you won’t be disappointed.

That’s the trouble with living things. Don’t last very long. Kittens one day, old cats the next. And then just memories. And the memories fade and blend and smudge together

I am going to be sharing my progress on the books going forward. It shouldn’t be considered as an indication of the quality of the book as my reading habits might vary depending on my workload. But it is usually a good indication of how interestingly the book progresses.

Use a custom job adapter in test

Mon, 17 Sep 2018 14:05:48 GMT

So you have decided to jump on board the ship to scalability with the custom job adapter from the previous post? Want to use this adapter in your tests? Do you use assert_enqueued_jobs in your tests? If yes, you wouldn’t see duplicate jobs being ignored from the queue. This is because it automatically sets the adapter to test_adapter irrespective of what you have in your test.rb. Not all is lost though. There is a way to use your custom adapter in tests by extending the ActiveJob::TestHelper.

# Specifies the queue adapter to use with all active job test helpers.
#
# Returns an instance of the queue adapter and defaults to
# <tt>ActiveJob::QueueAdapters::TestAdapter</tt>.
#
# Note: The adapter provided by this method must provide some additional
# methods from those expected of a standard <tt>ActiveJob::QueueAdapter</tt>
# in order to be used with the active job test helpers. Refer to
# <tt>ActiveJob::QueueAdapters::TestAdapter</tt>.
def queue_adapter_for_test
  ActiveJob::QueueAdapters::TestAdapter.new
end

Great. All we need is this piece of code in our test_helper and we should be good:

module ActiveJob::TestHelper
  def queue_adapter_for_test
    ActiveJob::QueueAdapters::UniqDelayedJobTestAdapter.new
  end
end

Not so fast, eh! Check out the docs:

Note: The adapter provided by this method must provide some additional methods from those expected of a standard ActiveJob::QueueAdapter in order to be used with the active job test helpers.

OK. A little more digging, and we can extend our adapter to support the methods that the test adapter requires

class ActiveJob::QueueAdapters::UniqDelayedJobTestAdapter < ActiveJob::QueueAdapters::UniqDelayedJobAdapter
  def enqueued_jobs
    Delayed::Job.all.to_a
  end

  def performed_jobs
    []
  end
end

Nice! Run the tests! Still failing… Looks like it expects the elements returned from enqueued_jobs to support a fetch method.

class Delayed::Backend::ActiveRecord::Job
  def fetch(key, default = nil)
    case key
    when :job
      return self.payload_object.job_data["job_class"] || default
    when :queue
      return self.queue || default
    else
      raise "cannot fetch #{key}. Not yet implemented"
    end
  end
end

Finally, we can use the adapter in tests. Don’t forget to update your expected job counts to account for the duplicate ones discarded by the new adapter!

Rae

Sat, 18 Aug 2018 06:02:48 GMT

Why am I in a dark box? Lynn remembered being drunk. She and Rox were partying hard. Their play had gone so well. It was a great story too. A story of power, greed and corruption. A story of how Rae, The Dark Lord, rose to godhood with treachery and lies. Born in a time of anarchy, a time when killing had no meaning, life had no value, a time when there was no religion. Rae was the saviour, a messiah, but not every hero’s path is devoid of Blood. He brought order to the people, no matter how dire the ways.

What did I do yesterday? I knew we should have stopped after Rox got sick. Damn, we were driving fast. Oh no, we were in an accident. And… blood.

Rox had died on the spot. Lynn had seen her best friend bleed to her death before collapsing herself.

I need to get out of here. They might have saved Rox.

“Hello… Someone please help. Help me!”, Lynn shouting now. The box was dark and small and somehow had enough oxygen that she hadn’t run out of breath yet.

“Hey, hey. Hold up. Praise Rae, you are finally up. ” A heavy voice from outside. Light started creeping in as the box gradually opened. Lynn instinctively closed her eyes from the blinding brightness. Everything went red with the light refracting through her eye lashes.

Where am I? She hadn’t seen a room this white before. White walls, white ceilings, white machines. Even the only other person in the room was wearing white.

“Welcome back. You are in the Northwestern Medical center. You were put to sleep in 2018 after a car accident. We operated on you yesterday. By Rae’s mercy, you are as good as new now.” said the person, as if reading her mind.

“You put me to sleep in 2018? What is that supposed to mean? What… year is this? ”

“Don’t worry, it’s all right. We have been reviving the old people this week. You were put to cryogenic sleep because we didn’t have the technology then. And then… And then… Ah well, you will see soon enough. Someone will be with you soon to explain everything.”

The world had changed in the last century. There was apparently a new religion; A dark(er) religion. One that had it’s origins even further back than any other known religion. This was what made it the most appealing. Apart from the fact that it didn’t shy away from advising to do what was necessary. This religion had one God, Rae, the Dark Lord. Some even called him the Fallen Angel and the Creator of Creators, but from where these names have come from, Lynn had no idea.

It all sounded so familiar to her. Most of it was eerily familiar, in fact, because she was the one to write it. She and Rox. Oh, how helpless Rox was, dying in a pool of her own blood. And Lynn did nothing, could do nothing but watch. She remembered being injured, her head bleeding a river of it’s own. And yet, here she was, practically unscratched in a world so similar, yet completely different.

This was all secondary though. There was something else, something about this new religion that didn’t sound right to her. She had seen huge temples on the way to the shelter, all dedicated to Rae. And the appearance of the mammoth statues on the top. She had described them before, seen them before. Somehow, the religion that she had made, a religion that she invented, was now the epicenter of everything.

This is my first attempt at fiction. I would be happy to hear your comments and feedback on this.

And do let me know if you want to hear this story further.

Remote testing and debugging of iOS apps

Wed, 01 Aug 2018 01:20:33 GMT

One problem that many iOS app developers have encountered is how to debug iOS programs being run on remote iOS devices. Good thing if you can connect an iPhone, iPad, iPod, etc. directly to your development computer. But what should you do if you don’t have the iOS device around to plug it into your local machine? Is there any way to get remote access to the device and applications it’s running?

Xcode to the rescue - debug iOS apps wirelessly

Thanks to the relatively new feature found in Xcode 9, iOS app developers are able to test their creations remotely. That means you don’t need to always physically connect and disconnect the device for app installation and debugging, you’re able to just access the required iOS gadget over the network.

However, before choosing this method you should note that:

In order to set up remote debugging, you’ll still need to physically connect the iOS device to your development machine.
Mac running Xcode and the iOS device you want to connect remotely must share the same local network. Otherwise, you won’t be able to use this feature.

FlexiHub: connect to remote iOS devices from anywhere

If the feature found in Xcode does not fit your purposes, you definitely should give a try to FlexiHub.

FlexiHub caught my eye precisely because the software works equally well across any network and is extremely simple to install and use.

FlexiHub’s unique port virtualization technology and its intuitive GUI are its strong points. The app is designed to virtualize iOS devices and redirect them over the network. That means you don’t need to have the target device near your local PC to be able to access its functionality and contents. This could be a significant advantage for mobile app developers.

Here’s how FlexiHub works:

When you first join the service, it suggests that you create an account and sign up for a free trial.
Then, install FlexiHub on the local Mac running the debugger as well as on the remote machine where the iOS device is connected physically.
After that, start the software on the local machine and log into the FlexiHub account.
Repeat the previous step for the remote computer.
In the software interface, you’ll see this device available for connection. All you need to do is click “Connect”.

After that, the remote iOS device will appear in the Device Manager of your local computer like it was directly connected to that machine.

Again, this is all about letting you go straight to app debugging on a remote device in the simplest and fastest way.

If FlexiHub is not exactly what you are looking for right now, here is a simple step-by-step instruction on how to enable remote debugging in Xcode:

First, you need to make sure that you’re running Xcode 9.0 or later, macOS 10.12.4 or later, and iOS 11.0 or later.
Then, to set up iPhone, iPad, or iPod touch, open your project on Xcode.
Choose “Window” > “Devices and Simulators”.
Click “Devices” in the invoked window.
Connect the required device to your Mac with the help of a USB cable.
In the left column, select the device and enable “Connect via network” in the detail area.
Now, Xcode will pair with your device. If the connection is established, you’ll see a network icon next to the device in the left column.
Finally, disconnect the device.

Here you go! Now, everything is set up to start the remote debug of your iOS app.

Better Background Jobs in Rails

Fri, 20 Jul 2018 15:00:00 GMT

Scalability is something that is deemed easily achievable in the current time. Heroku and AWS can give us that extra power required to scale our apps with a few clicks and we are good to go. But true scalability lies in code. One common approach to achieving fast response times is to delegate time-consuming tasks to a (group of) background worker(s) that can possibly work in parallel and knock off the heavy slabs taking the load off the web process.

What we usually overlook is the load these job workers need to handle. At our organization, we enqueue background jobs from a lot of after_save callbacks. When you reach a complex schema with objects saving their children, it’s hard to keep track of how many jobs even a small change could trigger. And a lot of times, most of these jobs are duplicate ones caused by the saving of the same object from different links.

How do we solve this? It can’t be easy tracking all those callbacks and prevent multiple jobs, right? The solution, as it turned out, was much simpler. A custom job adapter that could knock away those duplicate jobs keeping only one copy per job. We were able to decrease our job counts by more than 50% with this simple adapter. How did we make this adapter? Let’s check out the original delayed jobs adapter first:

Great, looks simple. All we need is to override enqueue and enqueue_at methods to find duplicate jobs and skip enqueueing them. If you study the format of the handler in the delayed_jobs table, it’s pretty easy to find duplicate jobs. All we need is to check for the job class and the passed arguments. Here’s an example of how simply this can be implemented.

def already_queued?
  query = Delayed::Job.where("handler ilike '%#{self.class}%'")
                      .where(:locked_at => nil, :locked_by => nil)
  query = update_query_for_arguments(query, arguments)
  query.any?
end

def update_query_for_arguments(query, arguments)
  arguments.each do |a|
    query = query.where("handler ilike '%#{a.try(:to_global_id) || a}%'")
  end
  query
end

I have left update_query_for_arguments to be very simple, but it can be extended to support array and hash arguments if you need it.

Finally, time for our shiny new adapter that would automatically discard the duplicate jobs coming in.

module ActiveJob
  module QueueAdapters
    class UniqDelayedJobAdapter < ActiveJob::QueueAdapters::DelayedJobAdapter
      class << self
        alias super_enqueue enqueue
        alias super_enqueue_at enqueue_at

        def enqueue(job)
          return if job.try(:already_queued?)
          super_enqueue(job)
        end

        def enqueue_at(job, timestamp)
          return if job.try(:already_queued?)
          super_enqueue_at(job, timestamp)
        end
      end
    end
  end
end

Finally, enable the adapter at a convenient location (e.g. application.rb or one of the environment specific initializers) and you are good to go:

config.active_job.queue_adapter = :uniq_delayed_job

Now you can enqueue to your job queue without worrying about duplicates eating up your hours for the worker.

MyJob.perform_later(my_arg) # Enqueued MyJob to delayed_jobs
MyJob.perform_later(my_arg) # No longer enqueues another job unless the previous one had already started processing

Update Localization on iOS at Runtime

Mon, 25 Dec 2017 08:19:24 GMT

iOS apps will run on many devices in many regions. To reach the most users, your application should handle text, audio files, numbers, currency, and graphics in ways appropriate to the locales where your application will be used. The framework automatically selects the resources that best match the device. Such a behavior is ok for most apps. But for some specialized apps, it might be required to change the localization at runtime, e.g. through in-app Settings.

In-app localization

On iOS, there are a lot of arguments as to why this is not a good pattern. Generally, from a programmatic point of view, it’s not easy if you are using Storyboards and there are a lot of hacks involved. Anyway, if this is something that you absolutely must include in your app, you are at the right place.

Loading localized string in code

Let’s get the simple things out of the way first. To load localized strings in code, you first need to load the bundle for the appropriate language.

let bundle: Bundle = .main
// Try to load from language specific bundle
if let path = bundle.path(forResource: "fr", ofType: "lproj"),
  let bundle = Bundle(path: path) {
  return bundle.localizedString(forKey: "Hello, World", value: nil)
} 
// Load from Base bundle
else if let path = bundle.path(forResource: LCLBaseBundle, ofType: "lproj"),
  let bundle = Bundle(path: path) {
  return bundle.localizedString(forKey: "Hello, World", value: nil)
}

Persist selected language

The next step that is pretty straightforward is to persist the user’s selection of language in UserDefaults.

UserDefaults.standard.set(selectedLanguage, forKey: "PGCurrentLanguageKey")
UserDefaults.standard.synchronize()

You need to read the USerDefaults later whenever the localization of a string is required.

func currentLanguage() {
  if let currentLanguage = UserDefaults.standard.object(forKey: "PGCurrentLanguageKey") as? String {
    return currentLanguage
  }
  return defaultLanguage()
}

The above two are so common, that there are already libraries available to handle this stuff for you. Localize-Swift is one such library that handles this and provides very clean helpers to help your localization needs. With it, you can write things like:

textLabel.text = "Hello World".localized()

Localize.setCurrentLanguage("fr")

Localize.resetCurrentLanguageToDefault()

Storyboard Localization

If you start using this, one thing that you will notice straight away is that it doesn’t help with Storyboard localization. Why? The initial locale on the Storyboard is read right after you load the storyboard. This could be done in your AppDelegate if you manually load the storyboard and set the window or automatically by UIKit if you have set the Storyboard in your project settings.

Restart app after language change

One possible solution for localized storyboards is to update the AppleLanguages key in UserDefaults when the user changes the language in the app and then force the user to restart the app.

NotificationCenter.default.addObserver(self, selector: #selector(languageUpdated), name: NSNotification.Name(LCLLanguageChangeNotification), object: nil)

func languageUpdated() {
  UserDefaults.standard.set(Localize.currentLanguage(), forKey: "AppleLanguages")
  UserDefaults.standard.synchronize()
  // TODO: Show alert asking user to restart the app for language change to take effect
}

Use stubs in storyboard

The second alternative is to just use stubs for all text in the storyboard and then set the text manually in code. This might be very cumbersome for even standard size apps, but is definitely the best way to go to allow a perfect flow for the user when he changes the language.

Reload Storyboard locally after a language update

This requires an extension on Bundle that tricks it into loading strings from the current localization instead of the cached locale that UIKit selects when starting the app. Create an extension on Bundle and override

func localizedString(forKey key: String, value: String?, table tableName: String?) -> String {
  let bundle: Bundle = bundle ?? .main
  if let path = bundle.path(forResource: Localize.currentLanguage(), ofType: "lproj"), let bundle = Bundle(path: path) {
    return bundle?.localizedString(forKey: key, value: value, table: tableName) ?? ""
  } else {
    return super.localizedString(forKey: key, value: value, table: tableName)
  }
}

This makes the Storyboard load strings from the updated locale for any view controllers that are constructed after the change. In order to change the existing views, you still need to either change them manually, reconstruct the current view controller (dismiss/pop and present/push again) or simple send him back to the original view controller of the app. In your AppDelegate, after a language change, do this:

var storybaord = UIStoryboard(name: "Main", bundle: nil)
window.rootViewController = storybaord.instantiateInitialViewController()

Login and Authenticated Sections to React/React-Native App

Sun, 12 Nov 2017 07:43:14 GMT

Most single page apps are likely to have some authenticated sections that should be inaccessible unless the user has identified itself. Adding authentication can be a non-trivial challenge for any app and though it’s not very difficult, there are several possible ways to approach this.

The most obvious and the arguably one that offers best security is to have the authentication separated from the app. But this either requires maintaining two different versions of the app for the authenticated and non-authenticated sections or requires server-side page flows. However, this is not always possible, for example when you want to have an installed app that needs to work offline.

This post describes an architecture using React, react-router to handle login and authenticated sections of an app.

With React Router, Route is just a component, just like div. So it is possible to nest multiple routes inside another Route as if it was a div. This makes it really simple to abstract all our authentication logic to one component that could allow/disallow the users to access all nested sections.

This means that you can layout your app like this

<App>
  <Switch>
    <Route path="/login" component={LoginPage} />
    <AuthenticatedRoutes />
  </Switch>
</App>

Simple enough, right?

It’s also really easy to implement the AuthenticatedRoutes component to redirect to login in case the user tries to access one of the nested pages. All of it can fit in 30 lines:

class AuthenticatedRoutes extends Component {
  componentDidMount() {
    const { isLoggedIn, redirectToLogin } = this.props;
    if (!isLoggedIn) {
      redirectToLogin();
    }
  }

  render() {
    const { isLoggedIn, sidebar } = this.props;
    if (isLoggedIn) {
      return (
        <div>
          <Header />
          <Sidebar />
          <section className="content">
            <Route path="/" exact component={IndexPage} />
            <Route path="/route1" exact component={Route1Page} />
            <Route path="/route2">
              <div>
                <Route path="/route2/nested1" component={Route2Nested1Page} />
              </div>
            </Route>
          </section>
        </div>
      );
    }
    return null;
  }
}

This will take care of redirecting to login each time this component is mounted when the user is not logged in. The redirectToLogin action takes care of this and again, it is very simple to write:

export function redirectToLogin() {
  return (dispatch: (action: actionType) => void) => {
    dispatch(push('/login'));
  };
}

The dynamic route philosophy from react router makes laying out the authentication logic really simple and is very easy to understand with most of the logic in a single component.

Resuming downloads on iOS

Sat, 28 Oct 2017 14:29:59 GMT

Introduction

With the increasing amount of data being used by mobile apps, it’s becoming common for most apps to download data to support offline functions. The downloaded data could be for additional textures for supporting complex games, ebooks for readers or music/video files for video/music players or companion guide apps that are becoming the de-facto standard for every big tourist attraction.

In the previous post, I described how to add simple file downloads in your app in less than 100 lines of code. Most often, the apps implementing their own download managers forget one very important aspect of this process. Pausing and resuming downloads. As the sizes of download packages increase, it becomes even more important to allow users to pause the download and resume later when they have better internet connectivity.

Downloading files on iOS

For the complete code to download files on iOS, check out the previous post. We get a downloadRequest when initiating a download with Alamofire. In order to allow users to cancel the download, all we need is to cancel the request using downloadRequest.cancel() when the user presses Cancel.

Save resume data

When a download is cancelled (either on user’s request or due to a network or internal error), Alamofire invokes the response handler where we can handle the response error and process resume data. Neither iOS nor Alamofire, automatically save this resume data to resume the download later. In order to support resuming downloads, the first step is to save the resume data in the cache so that this can be used later when initiating the download again.

Let’s see how to handle resume data in response block from Alamofire.

if let error = defaultDownloadResponse.error {
  log.warning("Download Failed with error - \(error)")
  self.onError(.Download)
  if let resumeData = defaultDownloadResponse.resumeData {
    Shared.dataCache.set(value: resumeData, key: self.url)
  }
  return
}

We will be using data cache from Haneke in this post to store the resume data. If you are using another library or have rolled out your own cache implementation, you need to change the occurrences of Shared.dataCache to fit your implementation.

Resume download from saved data

The next step is to provide this resume data when trying to resume the download.

First, fetch the resume data from the cache.

Shared.dataCache.fetch(key: url).onSuccess {resumeData in
  self.startDownload(with: resumeData)
}.onFailure {_ in
  self.startDownload()
}

Now provide this data when initiating the download. Alamofire makes this really simple. All we need is to add the additional resumingWith param when creating the request.

Alamofire.download(resumingWith: resumeData, to: destination)

Putting everything together

Let’s put everything together into our shiny new beginDownload method.

internal func beginDownload() {
  UIApplication.shared.isIdleTimerDisabled = true

  self.downloadProgressView?.startProgress(to: 0, duration: 0)

  if SYSTEM_VERSION_LESS_THAN(version: "10.0") || SYSTEM_VERSION_GREATER_THAN(version: "10.2") {
    Shared.dataCache.fetch(key: url).onSuccess {resumeData in
      self.startDownload(with: resumeData)
      }.onFailure {_ in
        self.startDownload()
    }
  } else {
    self.startDownload()
  }
}

private func startDownload(with resumeData: Data? = nil) {
  let destination: DownloadRequest.DownloadFileDestination = {temporaryURL, response in
    if let suggestedFileName = response.suggestedFilename {
      do {
        let directory = try Utils.tempDirectory()
        self.downloadedFilePath = (directory + suggestedFileName)
        if let downloadedFilePath = self.downloadedFilePath {
          if downloadedFilePath.exists { try self.downloadedFilePath?.deleteFile() }
          return (URL(fileURLWithPath: downloadedFilePath.rawValue), [.removePreviousFile, .createIntermediateDirectories])
        }
      } catch let e {
        log.warning("Failed to get temporary directory - \(e)")
      }
    }

    let (downloadedFileURL, _) = DownloadRequest.suggestedDownloadDestination()(temporaryURL, response)
    self.downloadedFilePath = Path(downloadedFileURL.absoluteString)
    return (downloadedFileURL, [.removePreviousFile, .createIntermediateDirectories])
  }

  if let resumeData = resumeData {
    downloadRequest = Alamofire.download(resumingWith: resumeData, to: destination)
  } else {
    downloadRequest = Alamofire.download(url, to:destination)
  }

  downloadRequest?.downloadProgress {progress in
    DispatchQueue.main.async {
      self.downloadProgressView?.startProgress(to: CGFloat(progress.fractionCompleted * 100), duration: 0.1)
    }
  }.response { defaultDownloadResponse in
      // TODO Handle cancelled error
      if let error = defaultDownloadResponse.error {
        log.warning("Download Failed with error - \(error)")
        self.onError(.Download)
        if let resumeData = defaultDownloadResponse.resumeData  {
          Shared.dataCache.set(value: resumeData, key: self.url)
        }
        return
      }
      guard let downloadedFilePath = self.downloadedFilePath else { return }
      log.debug("Downloaded file successfully to \(downloadedFilePath)")
      self.unzipFile(at: downloadedFilePath)
  }
}

That’s all you need for supporting pause and resume on your iOS apps. The next time you are integrating download in an app, check out this post and do a favor to your users by integrating the resume function.

Downloading Files in iOS

Mon, 18 Sep 2017 14:40:00 GMT

Introduction

Most often, the apps implementing their own download managers forget one very important aspect of this process. Pausing and resuming downloads. As the sizes of download packages increase, it becomes even more important to allow users to pause the download and resume later when they have better internet connectivity.

Downloading Files

iOS has a great support for file downloads using the Foundation’s NSURLConnection. An NSURLConnection object lets you load the contents of a URL by providing a URL request object. The interface for NSURLConnection is sparse, providing only the controls to start and cancel asynchronous loads of a URL request.

Alamofire

Given the huge developer community, there are several tools and libraries already built around to make your life easier. Rather than reinventing the wheel with boilerplate around NSURLConnection, let’s try to set up downloads using Alamofire which greatly simplifies the download tasks.

From the documentation, it’s very simple to initiate file downloads using Alamofire using Alamofire.download. To select a download destination, we need to use the DownloadFileDestination block to return the file url download should be stored at. Finally, for listening to download progress updates, we can add the downloadProgress block. One thing to take care of in the progress block is that it is invoked on the background thread, so if you want to make some changes to the UI (e.g. updating a progress view), use DispatchQueue.main.async to execute the updates on UI thread.

Let’s put everything together:

private func beginDownload() {
   UIApplication.shared.isIdleTimerDisabled = true

   self.downloadProgressView?.setProgress(value: 0, animationDuration: 0)
   
   let destination: DownloadRequest.DownloadFileDestination = {temporaryURL, response in
     if let suggestedFileName = response.suggestedFilename {
       do {
         let directory = try Utils.tempDirectory()
         self.downloadedFilePath = (directory + suggestedFileName)
         if let downloadedFilePath = self.downloadedFilePath {
           if downloadedFilePath.exists { try self.downloadedFilePath?.deleteFile() }
           return (URL(fileURLWithPath: downloadedFilePath.rawValue), [.removePreviousFile, .createIntermediateDirectories])
         }
       } catch let e {
         log.warning("Failed to get temporary directory - \(e)")
       }
     }
     
     let (downloadedFileURL, _) = DownloadRequest.suggestedDownloadDestination()(temporaryURL, response)
     self.downloadedFilePath = Path(downloadedFileURL.absoluteString)
     return (downloadedFileURL, [.removePreviousFile, .createIntermediateDirectories])
   }

   downloadRequest = Alamofire.download(url, to:destination).downloadProgress {progress in
     DispatchQueue.main.async {
         self.downloadProgressView?.setProgress(value: CGFloat(progress.fractionCompleted * 100), animationDuration: 0.1)
     }
   }.response { defaultDownloadResponse in
     // TODO: Handle cancelled error
     if let error = defaultDownloadResponse.error {
         log.warning("Download Failed with error - \(error)")
         self.onError(.Download)
         return
     }
     guard let downloadedFilePath = self.downloadedFilePath else { return }
     log.debug("Downloaded file successfully to \(downloadedFilePath)")
     // TODO: Handle downloaded file
   }
 }

Common File Utilities

I often use common utility functions to access directory locations throughout the app (This is using Path from FileKit, so make sure you add it in Podfile or Cartfile if you plan to use these utilities too).

class Utils {
  internal static func tempDirectory() throws -> Path {
    return try self.directoryInsideDocumentsWithName(name: "temp")
  }

  internal static func directoryInsideDocumentsWithName(name: String, create: Bool = true) throws -> Path {
   let directory = Path(NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0]) + name
   if create && !directory.exists {
     try directory.createDirectory()
   }
   return directory
  }
}

That’s all you need to integrate a simple download component in your app. To learn more about downloading files, check out the next post that describes how to add support for pausing and resuming the downloads without redownloading the already downloaded content.

Hyperion Cantos

Mon, 05 Jun 2017 11:04:58 GMT

This four-part masterpiece by Dan Simmons is one of the best series I have read. I had heard amazing reviews about this and this was right in the spot with the kind of books I like to read. I started reading the Hyperion Cantos series this year and I can’t recommend it enough.

The first book in the series, Hyperion, tells the story of seven pilgrims who must face the creature called Shrike in the Time Tombs, huge structures that move backward through time. It has masterful storytelling filled with lots of surprises, action, excitement, and sci-fi. The earth is destroyed and humanity is spread across hundreds of planets across the universe.

Most of the worlds are connected through Farcaster Portals (as big as huge arcs and as small as a door) that can teleport you from one planet to the other. So if you are really rich, you could have a home that has a bedroom looking over the ocean on Maui Covenant and the terrace opening into vast forests on God’s Grove.

Space travel is not dead, so you can travel through space in large spaceships, but be aware of the time debt accrued when traveling via Hawking Drive. What are a few months for you could be several years/decades for your family and friends.

Anyway, enough about the technology. The book tells the story of seven pilgrims on a journey to face the Shrike. The one who survives would get a wish from the Shrike.

The second book, The Fall of Hyperion picks up where the first one left. On the world of Hyperion, the mysterious Time Tombs are opening. And the secrets they contain mean that nothing—nothing anywhere in the universe—will ever be the same. This is a tale of treachery and humanity’s fight to survive. I cannot describe this in more details without spoiling the first book, but this in every way as good as the first one.

The third and fourth books are set a couple of centuries after the fall. These are not as much a continuation of the story that finished (or rather didn’t finish) with The Fall of Hyperion, but it comes full circle in the end and everything is wonderfully explained.

Endymion, the third part of the epic saga tells the story of love and memory, triumph and terror. It is set in a world where the cruciform has spread throughout the galaxy, has been tamed and promises everyone a life after death, immortality without any of the side effects that the Bikura went through.

It’s a journey of a messiah and her protector running from the religious missionaries along the (now) dead River Tethys. A story that finishes at the same place where the started, it will certainly leave you wondering and excited for the finale to come.

The epic finale, The Rise of Endymion follows the story of the messiah and her protector to new worlds, new dangers, and the Pax. Again, I can’t say more without spoiling it for you, but this is a fitting finale to the series which is guaranteed to leave you in tears. Very thrilling, filled with fun and surprises. A great tale about politics, AI, humanity and love.

Happy reading!

Audio Processing on iOS using Aubio

Sat, 06 May 2017 14:17:02 GMT

Aubio is a tool designed for the extraction of annotations from audio signals. Its features include segmenting a sound file before each of its attacks, performing pitch detection, tapping the beat and producing midi streams from live audio.

It comes prebuilt for iOS in form of a framework that can be just dragged into a Xcode project to get going. This post will touch some basics of working with the aubio framework on iOS. Let’s walk through a simple beat detection task with aubio.

The first step is the easiest

import aubio

To detect beats, we need to create a tempo detector. You need to pass a method, the buffer size, hop size (number of frames between two consecutive runs. A good value is usually buffer size / 2) and the sample rate.

let tempo: COpaquePointer? = new_aubio_tempo()

It’s usually a good idea to ignore long silences. It has a method to do just that that accepts the pointer to the tempo detector and a float silence threshold in dB.

aubio_tempo_set_silence(tempo!, silenceThreshold)

In order to perform detection, we need either to feed raw data to the detector.

If you want to run this on a file, it’s easy using an aubio source:

let samples = new_fvec(512)
let source = new_aubio_source( "/path/to/file.wav", 0, 512)
let out = new_fvec(1)
var read : uint_t = 0
while true {
    aubio_source_do(source, samples, &read )
    aubio_tempo_do(tempo, samples, out)
    if (fvec_get_sample(out, 0) != 0) {
        let beat_time : Float = Float(total_frames) / Float(samplerate)
        puts( String(format: "beat at %.2f", beat_time))
    }
    if (read < 512) {
        break
    }
}
del_fvec(out)
del_aubio_tempo(tempo)
del_aubio_source(source)
del_fvec(samples)

But who wants to run on a file, it boring! aubio is optimized to even run the algorithms on real-time audio coming from the microphone. First step is to get raw data from the microphone. There’s a really nice gist to get this done.

Use this, but update the processMicrophoneBuffer to feed the data to aubio:

func setupAubio(samplerate: UInt32) {
    samples = new_fvec(sampleSize)
    tempo = new_aubio_tempo("default", 1024, sampleSize, samplerate)
    aubio_tempo_set_silence(tempo!, silenceThreshold)
}

func processMicrophoneBuffer(inputDataList : UnsafeMutablePointer<AudioBufferList>, frameCount : UInt32) {
    guard let samples = samples, tempo = tempo else { return }
    let out = new_fvec(2)
    var sampleCount: UInt32 = 0
    for i in 0..<(count/2) {
        let x = Float(dataArray[i+i  ])   // copy left  channel sample
        let y = Float(dataArray[i+i+1])   // copy right channel sample
        
        fvec_set_sample(samples, x*x + y*y, sampleCount)
        sampleCount += 1
        if sampleCount == sampleSize || i == count/2-1 {
            aubio_tempo_do(onset, samples, out)
            if (fvec_get_sample(out, 0) != 0) {
                // Yay! A BEAT!!!
                break
            }
            sampleCount = 0
        }
    }
    del_fvec(out)
}

func stopRecording() {
    if let tempo = tempo, samples = samples {
        del_aubio_tempo(tempo)
        del_fvec(samples)
        self.tempo = nil
        self.samples = nil
    }
}

Note: Make sure you clean up the pointers in stopRecording to avoid memory leaks.

iOS Music Player on the Lock Screen

Fri, 05 Aug 2016 08:57:30 GMT

Building a music player app and want the system to display the current track information on the device lock screen and in the multimedia controls? You can use a now playing info center to set now-playing information for media being played by your app. This takes care of showing the music info and playback controls on the lock screen. If the user directs playback of your media to Apple TV via AirPlay, the now-playing information appears on the television screen. If the user connects a device to an iPod accessory, such as in a car, the accessory may display now-playing information.

MPNowPlayingInfoCenter is the class that controls this information. The information you can specify includes a subset of the properties available in the media item class (MPMediaItem), as well as some properties specific to this class. You do not have direct control over which information is displayed, or its formatting. You set the values of the now playing info center dictionary according to the information you want to provide to the system. The system, or the connected accessory, handles the information’s display in a consistent manner for all apps.

First, let’s discuss the setup required to register our app as a publisher for now playing information. The first requirement is to start receiving remote control events so the users can control the music from the playback controls on the lock screen. Next, we need to listen to the controls we will be handling and perform functions on our app accordingly based on these remote control events.

private func setupNowPlayingInfoCenter() {
    UIApplication.sharedApplication().beginReceivingRemoteControlEvents();
    MPRemoteCommandCenter.sharedCommandCenter().playCommand.addTargetWithHandler {event in
        self.audioPlayer.resume()
        self.updateNowPlayingInfoCenter()
        return .Success
    }
    MPRemoteCommandCenter.sharedCommandCenter().pauseCommand.addTargetWithHandler {event in
        self.audioPlayer.pause()
        return .Success
    }
    MPRemoteCommandCenter.sharedCommandCenter().nextTrackCommand.addTargetWithHandler {event in
        self.next()
        return .Success
    }
    MPRemoteCommandCenter.sharedCommandCenter().previousTrackCommand.addTargetWithHandler {event in
        self.prev()
        return .Success
    }
}

The next part is actually publishing the information about the current track so that it can be displayed on the UI. Here’s a simple function that I use to update the information whenever a track changes or some important event occurs in the music player.

private func updateNowPlayingInfoCenter(artwork: UIImage? = nil) {
    guard let file = currentItem else {
        MPNowPlayingInfoCenter.defaultCenter().nowPlayingInfo = [String: AnyObject]()
        return
    }
    if let imageURL = file.album?.imageUrl where artwork == nil {
        Haneke.Shared.imageCache.fetch(URL: imageURL, success: {image in
            self.updateNowPlayingInfoCenter(image)
        })
        return
    }
    MPNowPlayingInfoCenter.defaultCenter().nowPlayingInfo = [
        MPMediaItemPropertyTitle: file.title,
        MPMediaItemPropertyAlbumTitle: file.album?.title ?? "",
        MPMediaItemPropertyArtist: file.album?.artist?.name ?? "",
        MPMediaItemPropertyPlaybackDuration: audioPlayer.duration,
        MPNowPlayingInfoPropertyElapsedPlaybackTime: audioPlayer.progress
    ]
    if let artwork = artwork {
        MPNowPlayingInfoCenter.defaultCenter().nowPlayingInfo?[MPMediaItemPropertyArtwork] = MPMediaItemArtwork(image: artwork)
    }
}

There are several metadata properties that you can specify on the now playing info center and it is recommended to set as many as possible. Note that here we use Haneke for loading images from the network. In case we don’t have a cached image straight away, we can set the information without the artwork and then queue the updates again when we fetch the image.

Building a music player for iOS

Sat, 23 Jul 2016 10:32:52 GMT

There are already a few tutorials about building building an app that plays music on iOS. Why am I writing a whole new post about it then? Most of the existing tutorials focus on the really basic concepts about a music player app. This is going to be different. We will set up a full fledged music player with it’s own music library, ability to create custom playlists, streaming songs online etc.

This is not a post about accessing the existing music on user’s device, so if you are looking for that, feel free to jump back to Google and resume your search. This also isn’t about the low level use of MediaPlayer framework. We will be using a library to handle this.

Building the Music Player

The first thing we are going to tackle in the post is creating the Music Player. We are using StreamingKit, so we have most of our work cut for us, but it still requires a bit of handling logic around it to implement all the features that a Music Player of this generation needs.

Initialization

To play music with StreamingKit, we need a STKAudioPlayer. Check out all the options for the audio player in the source.

var audioPlayer: STKAudioPlayer!

// MARK: Init
override init() {
    super.init()
    resetAudioPlayer()
}

private func resetAudioPlayer() {
    var options = STKAudioPlayerOptions()
    options.flushQueueOnSeek = true
    options.enableVolumeMixer = true
    audioPlayer = STKAudioPlayer(options: options)
    
    // Set up audio player
    audioPlayer.meteringEnabled = true
    audioPlayer.volume = 1
    audioPlayer.delegate = self
}

Handling a queue

Next thing a music player needs is to handle a queue of music files. You could get away with handling just one file per player and simplify the logic, but since we are focusing on a full fledged music player, we will see how to handle the queue. We create a class Track to handle the properties of each queue element. I will be skipping the implementation since it would be pretty specific depending on the app you are working on, but the main properties it should be supporting are identifier and a url (the URL for the music file, local/streaming).

private(set) var queue: [AlbumFile]

// MARK: Public API
internal func playWithQueue(queue: [AlbumFile], index: Int = 0) {
    guard index >= 0 && index < queue.count else {
        return
    }
    self.queue = queue
    audioPlayer.clearQueue()
    if let url = queue[index].url {
        audioPlayer.playURL(url)
    }
    for i in 1 ..< queue.count {
        audioPlayer.queueURL(queue[Int((index + i) % queue.count)].url ?? NSURL())
    }
    currentIndex = index
    loop = false
}

Controlling the playback

While we are adding a public API to the Music Player, let’s also add simple helper functions to control the playback.

internal func stop() {
    audioPlayer.stop()
    queue = []
    currentIndex = -1
}

internal func play(file: AlbumFile) {
    audioPlayer.playURL(file.url ?? NSURL())
}

internal func next() {
    guard queue.count > 0 else {
        return
    }
    currentIndex = (currentIndex + 1) % queue.count
    playWithQueue(queue, index: currentIndex)
}

internal func prev() {
    currentIndex = max(0, currentIndex - 1)
    playWithQueue(queue, index: currentIndex)
}

Handling audio player events

STKAudioPlayer communicates several events during playback to its delegate like when an item starts playing, finishes buffering, finishes playing or encounters an error. Let’s see how we can handle these events to perform meaningful operations in the music player.

A track starts playing: Update the local state to be communicated later to the UI
A track finishes buffering: Update info in the now playing center (check out the next post to find out how to interface with the now playing center)
Audio player state changes: Update the local state to be communicated later to the UI
A track finishes playing: Switch to the next track
Unexpected error occurs: Reset and try to play again or inform the user about the error depending on your app requirements.

// MARK: Audio Player Delegate
func audioPlayer(audioPlayer: STKAudioPlayer, didStartPlayingQueueItemId queueItemId: NSObject) {
    if let index = (queue.indexOf { $0.url == queueItemId }) {
        currentIndex = index
    }
}

func audioPlayer(audioPlayer: STKAudioPlayer, didFinishBufferingSourceWithQueueItemId queueItemId: NSObject) {
    updateNowPlayingInfoCenter()
}

func audioPlayer(audioPlayer: STKAudioPlayer, stateChanged state: STKAudioPlayerState, previousState: STKAudioPlayerState) {
    self.state.value = state
    if state != .Stopped && state != .Error && state != .Disposed {
    }
    updateNowPlayingInfoCenter()
}

func audioPlayer(audioPlayer: STKAudioPlayer, didFinishPlayingQueueItemId queueItemId: NSObject, withReason stopReason: STKAudioPlayerStopReason, andProgress progress: Double, andDuration duration: Double) {
    if let index = (queue.indexOf { $0.url == audioPlayer.currentlyPlayingQueueItemId() }) {
        currentIndex = index
    }
    
    if stopReason == .Eof {
        next()
    } else if stopReason == .Error {
        stop()
        resetAudioPlayer()
    }
}

func audioPlayer(audioPlayer: STKAudioPlayer, unexpectedError errorCode: STKAudioPlayerErrorCode) {
    print("Error when playing music \(errorCode)")
    resetAudioPlayer()
    playWithQueue(queue, index: currentIndex)
}

Supporting background audio

In order to play music while your app is in the background, you must do some additional set up.

Update your Info.plist file to include the audio in the UIBackgroundModes array.
Initialize your AudioSession somewhere near the beginning of your app. A good point to do this is in AppDelegate.application(application:, didFinishLaunchingWithOptions:).

func setupMusicPlayer() {
    do {
        try AVAudioSession.sharedInstance().setCategory(AVAudioSessionCategoryPlayback)
    } catch {
        print("Failed to setup audio")
    }
}

Playing music

After we wrap everything up in a class (named MusicPlayer for the tutorial), we are good to go. Since you would normally be supporting just one instance of the music player in your app, it’s a good idea to create a singleton and use it all around your app. Here’s how you can set it up.

class MusicPlayer: NSObject, STKAudioPlayerDelegate {
    private static var instance = MusicPlayer()
    // …
    
    // MARK: Static
    internal static func sharedInstance() -> MusicPlayer {
        return instance
    }
}

Here’s how you can use it from other parts of the UI.

MusicPlayer.sharedInstance().playWithQueue(tracks, index: indexPath.row)

Why you should start using RxSwift for iOS

Wed, 06 Jul 2016 14:39:30 GMT

So, you have heard about RxSwift. You see it ever so frequently in the wild. But whenever you try to wrap your head around it, it hurts. Here’s what the Github page says about RxSwift:

Rx is a generic abstraction of computation expressed through Observable interface. This is a Swift version of Rx.

So great, you go and start watching that video. It’s great, it is going to change programming for ever, but how do I use it. Let’s try to break it down. RxSwift is an implementation of ReactiveX. In the most simple terms, the ReactiveX Observable model allows you to treat streams of asynchronous events with the same sort of simple, composable operations that you use for collections of data items like arrays. No more tangled webs of callbacks.

When building a frontend application, most of your tasks involve reacting to user interactions and doing something in response to them. It’s a great use case for RxSwift. Let’s take the example of a simple log in flow. Here are the events that you usually need to take care of:

Make a backend call to authenticate the user when he clicks login.
On success, get user’s info from the backend.
Handle errors and show messages on UI.

APIProvider.request(.Login(emailTextField.trimmedText, passwordTextField.trimmedText)).flatMapLatest { token in
    return APIProvider.request(.UserAccount)
}.subscribe { event in
    switch event {
    case .Next(let user):
        log.debug("Login success: \(user.user.email)")
        self.loginDidFinishWithUser(user.user)
    case .Error(let error):
        loginDidFailWithError(error)
    default:
        break
    }
}.addDisposableTo(disposeBag)

Let’s break this up.

.Login is the actual call to the backend to authenticate our user. You could use any implementation that you prefer (I have used Moya that plays really nice with RxSwift) and returns an Observable.
If the login succeeds, it calls flatMap with an item of the observable. flatMap returns another observable which could then be subscribed and observed again using chaining. This is where we make another request to the backend for the UserAccount if our login was successful.

Finally, with .subscribe, we start listening to the events from this Observable chain. The event could be .Next when a new element is emitted by the Observable. This means that the request was successful. It could be .Error in case of an error and .Completed when the observable completes.

Let’s assume we want to consider failure status codes as .Error events on the observable. This becomes really easy with RxSwift. All we need is a filterSuccessfulStatusAndRedirectCodes from Moya in the chain after our request. If you want to map the response into Swift objects, it’s got you covered there as well. Just use .mapObject(Class) from Moya-ObjectMapper in the chain. You see how easy it was to make changes later by just adding things in the Observable chain?

This is a very simple example of how to manage simple network calls in your application with RxSwift. It requires a bit of getting used to, but it’s well worth the effort. It helps write maintainable code that is easy to read or refactor at a later stage. If at any point, we want to add another call to the chain, we would simply need to add another flatMap operator in the Observable chain.

This is supposed to be a very simple primer to RxSwift and in no way even close to the covering the myriad of operators available at your convenience.

Why you should start using RxJava on Android

Fri, 24 Jun 2016 16:07:15 GMT

So, you have heard about RxJava. You see it ever so frequently in the wild. But whenever you try to wrap your head around it, it hurts. Here’s what the Github page says about RxJava:

RxJava is a Java VM implementation of Reactive Extensions: a library for composing asynchronous and event-based programs by using observable sequences.

It extends the observer pattern to support sequences of data/events and adds operators that allow you to compose sequences together declaratively while abstracting away concerns about things like low-level threading, synchronization, thread-safety and concurrent data structures.

That’s a mouthful. Let’s try to break it down. RxJava is an implementation of ReactiveX. In the most simple terms, the ReactiveX Observable model allows you to treat streams of asynchronous events with the same sort of simple, composable operations that you use for collections of data items like arrays. No more tangled webs of callbacks.

When building a frontend application, most of your tasks involve reacting to user interactions and doing something in response to them. It’s a perfect use case for RxJava. Let’s take the example of a simple log in flow. Here are the events that you usually need to take care of:

Make a backend call to authenticate the user when he clicks login.
On success, get user’s info from the backend.
Handle errors and show messages on UI.

BackendService.login(email, password)
  .subscribeOn(Schedulers.io()).observeOn(AndroidSchedulers.mainThread())
  .flatMap({ response ->
    if (response.isSuccess) {
      return@Func1 BackendService.userAccount().subscribeOn(Schedulers.io())
    }
    handleError(response)
    Observable.error<Response<User>>(Exception("Failed to get access token"))
  })
  .subscribeOn(Schedulers.io()).observeOn(AndroidSchedulers.mainThread())
  .subscribe({ response ->
    if (response.isSuccess) {
      onUserLoginSuccess(user, device)
    } else {
      handleError(response)
    }
  }, { throwable ->
    handleError()
  })

Let’s break this up.

.login makes the actual call to the backend to authenticate our user. You could use any implementation that you prefer (I have used Retrofit that plays really nice with RxJava) that returns an Observable. The implementation itself doesn’t need to worry about the thread it should perform the task on.
.subscribeOn instructs the observable to perform the task on a IO thread that performs the task in the background.
.observeOn instructs which thread the Observer will be observed upon. This is so that we can perform the UI tasks on the main thread.

If the login succeeds, it calls flatMap with an item of the observable. flatMap returns another observable which could then be subscribed and observed again using chaining. This is where we make another request to the backend if our login was successful.

Finally, with .subscribe, we start listening to the results from this Observable chain. The first argument to subscribe is invoked for each successful result of the Observable. If the Observable fails with an error, the second function is called instead with an Exception.

Let’s assume we did some changes to the UI when the login failed. Now we would like to clear UI errors when text fields are edited. RxJava makes it very simple (with RxBinding that converts UI interaction into RxJava Observables):

Observable.merge(emailEditText.textChanges(), passwordEditText.textChanges())
  .subscribe {
    updateButton(Progress.NORMAL)
  }

We simply get the .textChanges Observables for the edit texts, merge them into one Observable (which means that the subscribe will be called for events from either of these two Observables) and then reset the UI inside subscribe.

This is a very simple example of how to manage simple UI flows in your application with RxJava. It requires a bit of getting used to, but it’s well worth the effort. It helps write maintainable code that is easy to read or refactor at a later stage. If at any point, we want to add another call to the chain, we would simply need to add another flatMap operator in the Observable chain.

This is supposed to be a very simple primer to RxJava and in no way even close to the covering the myriad of operators available at your convenience.

Swift constructs for Kotlin

Fri, 13 May 2016 15:19:45 GMT

If you have used Swift for iOS development, chances are that you really miss its flexibility when developing for Android with Java. Kotlin brings all the Swift goodness to Java that combines OO and functional features and is focused on interoperability, safety, clarity and tooling support. Let’s see a few Swift constructs and their equivalent on Kotlin:

Unwrapping

In Swift, we unwrap with guard let

guard let unwrapped = wrapped else {
  return
}

In Kotlin, we use the elvis operator

val unwrapped = wrapped ?: return

Blocks and Lambdas

In Swift, we can reference parameters without specifying the complete signature using $0, $1, …

let doubleNumbers = numbers.map { $0 * 2 }

In Kotlin, we use it

val doubleNumbers = numbers.map { it * 2 }

Named Parameters

Swift uses named parameters

addNumber(1, with: 2) // Call with named parameters

func addNumber(number: Int, with: Int) {
  return number + with
}

Kotlin has named parameters too

addNumber(1, with = 2) // Call with named parameters

fun addNumber(number: Int, with: Int): Int = number + with

Computed Properties

We define computed properties with (optional) getter in Swift

var name: String {
  return "\(firstName) \(lastName)"
}

Computed properties defined with get in Kotlin

val name: String
  get() {
    return "$firstName $lastName"
  }

Or even smaller with = syntax

val name: String
  get() = "$firstName $lastName"

String templates

We write string templates like "\(firstName) \(lastName)" in Swift and "$firstName $lastName" in Kotlin

Add a circle around UIImage

Sun, 24 Jan 2016 09:00:30 GMT

For an app I worked on recently, we had the need to add a circle around an UIImage. When using a UIImageView, it is pretty straightforward to add a border to the image view. Unfortunately, we were using a library with no control over the UIImageView for a particular feature within the app and had to switch to manually adding the border in the UIImage before passing it over to the library. Here’s a category on UIImage to do so.

@implementation UIImage (PGHelpers)

- (UIImage *)borderedImageWithPadding:(int)padding {
    // Draw image
    UIGraphicsBeginImageContextWithOptions(CGSizeMake(self.size.width + padding, self.size.height + padding), NO, [UIScreen mainScreen].scale);
    CGContextRef context = UIGraphicsGetCurrentContext();
    [self drawInRect:CGRectMake(padding / 2, padding / 2, self.size.width, self.size.height)];

    // Add border
    UIBezierPath *bezier = [UIBezierPath bezierPath];
    [bezier setLineWidth:4.f];
    [bezier setLineJoinStyle:kCGLineJoinRound];
    [bezier addArcWithCenter:CGPointMake(self.size.width / 2 + padding / 2, self.size.height / 2 + padding / 2) radius:self.size.height / 2 + padding / 2 - 2 startAngle:0 endAngle:M_PI clockwise:NO];
    [bezier addArcWithCenter:CGPointMake(self.size.width / 2 + padding / 2, self.size.height / 2 + padding / 2) radius:self.size.height / 2 + padding / 2 - 2 startAngle:M_PI endAngle:2 * M_PI clockwise:NO];
    CGContextSetStrokeColorWithColor(context, [UIColor blackColor].CGColor);
    [bezier stroke];

    UIImage *result = UIGraphicsGetImageFromCurrentImageContext();
    UIGraphicsEndImageContext();
    return result;
}

@end

Create UIImage with text

Sat, 19 Dec 2015 12:53:58 GMT

While making an application, showing user’s image is a very common use case. Gone were the days when we used to show anonymous image in case a person’s profile image or display image is not present. We can easily show a person’s initials to make it easier to identify the person. Here’s a simple category to generate an UIImage with centered text overlayed on top of it.

@implementation UIImage (PGHelpers)
- (UIImage *)imageWithCenteredText:(NSString *)text withfont:(UIFont *)font {
    CGSize size = [text sizeWithAttributes:@{NSFontAttributeName : font}];
    return [self imageWithText:text atPoint:CGPointMake((self.size.width - size.width) / 2, (self.size.height - size.height) / 2) withFont:font];

}

- (UIImage *)imageWithText:(NSString *)text atPoint:(CGPoint)point withFont:(UIFont *)font {
    UIGraphicsBeginImageContextWithOptions(self.size, YES, 0.0f);
    [self drawInRect:CGRectMake(0, 0, self.size.width, self.size.height)];
    CGRect rect = CGRectMake(point.x, point.y, self.size.width, self.size.height);
    [[UIColor whiteColor] set];

    NSDictionary *att = @{NSFontAttributeName : font, NSForegroundColorAttributeName : [UIColor whiteColor]};
    [text drawInRect:rect withAttributes:att];

    UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
    UIGraphicsEndImageContext();

    return newImage;
}
@end

This can be chained with other methods (e.g. from UIImage-Additions, UIImage-ResizeMagick etc.) to create compelling profile images with advance blurring effects etc. Here’s an example that resizes the image to size x size, blurs it, adds the user initials in the center, crops it to a circle and finally adds a border around it:

UIImage *blurredImage = [[image resizedImageByMagick:[NSString stringWithFormat:@"%dx%d#", size * 3, size * 3]] blurredImageWithRadius:size / 8 iterations:3 tintColor:[UIColor clearColor]];
[[[blurredImage imageWithCenteredText:profile.avatarInitials withfont:[UIFont fontWithName:kPGFontNameAvenirNextCondensedRegular size:60]] add_imageWithRoundedBounds] borderedImageWithPadding:30];

Resizing images picked from UIImagePickerController

Sun, 29 Nov 2015 15:46:10 GMT

There’s already a lot of stuff about image resizing on iOS already available on the web with each discussing about potential pros and cons of the technique. There is also a good post on NSHipster summarizing some of the best available image resizing techniques.

With iOS 8, Apple has given us PhotoKit, a modern framework for managing photos without diving into the low level details and more performant than AssetsLibrary. Let’s see how we can resize a simple image selected using the UIImagePickerController using PhotoKit.

imagePickerController.bk_didFinishPickingMediaBlock = {
    (controller imagePickerController: UIImagePickerController!, dictionary: [NSObject:AnyObject]!) -> Void in
    imagePickerController.dismissViewControllerAnimated(true, completion: nil)

    guard let assetURL = dictionary[UIImagePickerControllerReferenceURL] as? NSURL else {
        if let image = dictionary[UIImagePickerControllerOriginalImage] as? UIImage {
            // TODO Image taken from the camera. Resize using one of the techniques discussed on https://nshipster.com/image-resizing/
        }
        return
    }

    // Fetch Photos Asset and resize image
    let fetchResult = PHAsset.fetchAssetsWithALAssetURLs([assetURL], options: nil)
    let options = PHImageRequestOptions()
    options.synchronous = false
    options.resizeMode = .Fast
    options.deliveryMode = .HighQualityFormat

    guard let asset = fetchResult.firstObject as? PHAsset else {
        if let image = dictionary[UIImagePickerControllerOriginalImage] as? UIImage {
            // TODO Cannot find the associated PhotoKit asset. Resize using one of the techniques discussed on https://nshipster.com/image-resizing/
        }
        return
    }

    // Request resized image
    PHImageManager.defaultManager().requestImageForAsset(asset, targetSize: CGSize(width: 1024, height: 1024), contentMode: .AspectFit, options: options, resultHandler: { (image, _) -> Void in
        // image is the resized image
    })
}

The code is pretty straightforward. We fetch the PHAsset from the asset url of the image and then use the PHImageManager to get the resized image. There are a few options to control the scaling mode.

deliveryMode : Controls the requested image quality and delivery priority.
Opportunistic: Photos automatically provides one or more results in order to balance image quality and responsiveness. The resultHandler block more than once with different quality images as they become available.
HighQualityFormat: Photos provides only the highest-quality image available, regardless of how much time it takes to load.
FastFormat: Photos provides only a fast-loading image, possibly sacrificing image quality.
resizeMode: A mode that specifies how to resize the requested image.
None: Photos does not resize the image asset.
Fast: Photos efficiently resizes the image to a size similar to, or slightly larger than, the target size.
Exact: Photos resizes the image to match the target size exactly.

Show avatar on both sides for Atlas Conversation View Controller (Layer Chat)

Sat, 11 Jul 2015 09:27:50 GMT

Atlas is a fully featured, high performance, 100% customizable UI kit, built by Layer to power communications interfaces in any app. If you want to learn how to integrate chat in your iOS app, take a look at Implementing Chat/Messaging in iOS apps with Layer.

Atlas does provide quick customization of some common options. But it also assumes some pretty rigid defaults which are not easy to change unless you are willing to spend some time digging through it’s codebase. One such thing is when you want to show avatar on both sides of the conversation. By default, the ÀTLConversationViewController only allows showing avatar on the incoming messages.

Add avatars on Outgoing Messages

To add avatars on outgoing messages, we need to subclass the ATLMessageCollectionViewCell and override its methods to make the avatar visible. There are three steps here:

Update the constraints

@implementation PGOutgoingMessageCollectionViewCell

NSString *const PGOutgoingMessageCellIdentifier = @"PGOutgoingMessageCellIdentifier";
CGFloat const PGAvatarPGAvatarImageRightPadding = 12.0f;
CGFloat const PGAvatarImageLeftPadding = 7.0f;

- (id)initWithFrame:(CGRect)frame {
    self = [super initWithFrame:frame];
    if (self) {
        [self lyr_incommingCommonInit];
    }
    return self;
}

- (id)initWithCoder:(NSCoder *)aDecoder {
    self = [super initWithCoder:aDecoder];
    if (self) {
        [self lyr_incommingCommonInit];
    }
    return self;
}

- (void)lyr_incommingCommonInit {
    [self configureConstraintsForIncomingMessage];
}

- (void)configureConstraintsForIncomingMessage {
    [self.avatarImageView mas_makeConstraints:^(MASConstraintMaker *make) {
        make.right.equalTo(self.contentView.mas_right).with.offset(-PGAvatarImageRightPadding);
        make.bottom.equalTo(self.avatarImageView.mas_bottom);
        make.height.equalTo(@35);
        make.width.equalTo(@35);
    }];
    self.bubbleWithAvatarRightConstraint = [NSLayoutConstraint constraintWithItem:self.bubbleView attribute:NSLayoutAttributeRight relatedBy:NSLayoutRelationEqual toItem:self.avatarImageView attribute:NSLayoutAttributeLeft multiplier:1.0 constant:-PGAvatarImageLeftPadding];
    [self.contentView addConstraint:self.bubbleWithAvatarRightConstraint];
    self.bubbleWithoutAvatarRightConstraint = [NSLayoutConstraint constraintWithItem:self.bubbleView attribute:NSLayoutAttributeRight relatedBy:NSLayoutRelationEqual toItem:self.contentView attribute:NSLayoutAttributeRight multiplier:1.0 constant:ATLMessageCellHorizontalMargin];
}

- (void)layoutSubviews {
    [super layoutSubviews];
    self.avatarImageView.layer.cornerRadius = CGRectGetHeight(self.avatarImageView.frame) / 2;
}
@end

Override display logic

- (void)shouldDisplayAvatarItem:(BOOL)shouldDisplayAvatarItem {
    NSArray *constraints = [self.contentView constraints];
    if (shouldDisplayAvatarItem) {
        if ([constraints containsObject:self.bubbleWithAvatarRightConstraint]) return;
        [self.contentView removeConstraint:self.bubbleWithoutAvatarRightConstraint];
        [self.contentView addConstraint:self.bubbleWithAvatarRightConstraint];
    } else {
        if ([constraints containsObject:self.bubbleWithoutAvatarRightConstraint]) return;
        [self.contentView removeConstraint:self.bubbleWithAvatarRightConstraint];
        [self.contentView addConstraint:self.bubbleWithoutAvatarRightConstraint];
    }
    [self setNeedsUpdateConstraints];
}

- (void)updateWithSender:(id <ATLParticipant>)sender {
    if (sender) {
        self.avatarImageView.hidden = NO;
        self.avatarImageView.avatarItem = sender;
    } else {
        self.avatarImageView.hidden = YES;
    }
}

Register our class for outgoing messages

In your ATLConversationViewController subclass, do the following somewhere in viewDidLoad

- (void)viewDidLoad {
    [self registerClass:[PGOutgoingMessageCollectionViewCell class] forMessageCellWithReuseIdentifier:PGOutgoingMessageCellIdentifier];
}

- (NSString *)conversationViewController:(ATLConversationViewController *)viewController reuseIdentifierForMessage:(LYRMessage *)message {
    if ([message.sentByUserID isEqualToString:self.layerClient.authenticatedUserID]) {
        return PGOutgoingMessageCellIdentifier;
    }
    return ATLIncomingMessageCellIdentifier;
}

Need help setting this up or just want to chat? Shoot me an email at pulkit110@gmail.com.

Add Click Handler to Avatars on Atlas (Layer Chat)

Sun, 21 Jun 2015 13:50:59 GMT

In the previous post, I talked about subclassing ATLMessageCollectionViewCell to display avatars on both sides in a conversation. Let’s take that a step further and see how we can handle tap on avatars in a conversation. We will subclass the same class this time, but for the incoming messages rather than the outgoing ones.

Create a cell for incoming messages

@interface PGIncomingMessageCollectionViewCell ()
@property(nonatomic, strong) UITapGestureRecognizer *avatarTapGestureRecognizer;
@end

@implementation PGIncomingMessageCollectionViewCell

NSString *const PGIncomingMessageCellIdentifier = @"PGIncomingMessageCellIdentifier";

- (void)lyr_incommingCommonInit {
    [super lyr_incommingCommonInit];

    self.avatarTapGestureRecognizer = [[UITapGestureRecognizer alloc] bk_initWithHandler:^(UIGestureRecognizer *sender, UIGestureRecognizerState state, CGPoint location) {
        // TODO Handle a tap on the avatar item
    }];
    [self.avatarImageView addGestureRecognizer:self.avatarTapGestureRecognizer];
    self.avatarImageView.userInteractionEnabled = YES;

    [self.avatarImageView mas_makeConstraints:^(MASConstraintMaker *make) {
        make.height.equalTo(@35);
        make.width.equalTo(@35);
    }];
}

- (void)layoutSubviews {
    [super layoutSubviews];
    self.avatarImageView.layer.cornerRadius = CGRectGetHeight(self.avatarImageView.frame) / 2;
}

@end

Register our cell for incoming messages

In your ATLConversationViewController subclass, do the following:

- (void)viewDidLoad {
    [self registerClass:[PGIncomingMessageCollectionViewCell class] forMessageCellWithReuseIdentifier:PGIncomingMessageCellIdentifier];
}

- (NSString *)conversationViewController:(ATLConversationViewController *)viewController reuseIdentifierForMessage:(LYRMessage *)message {
    if ([message.sentByUserID isEqualToString:self.layerClient.authenticatedUserID]) {
        return PGOutgoingMessageCellIdentifier;
    }
    return PGIncomingMessageCellIdentifier;
}

Need help setting this up or just want to chat? Shoot me an email at pulkit110@gmail.com.

Open full image in conversation controller with Atlas (Layer Chat)

Wed, 13 May 2015 19:45:36 GMT

By default, Atlas doesn’t open full resolution images when a message with an image is clicked. This tutorial explains how to configure Atlas to show the full image when on message click.

Atlas provides a conversationViewController:didSelectMessage method which the class implementing ATLConversationViewControllerDelegate can implement to perform actions when a message is clicked. We will check whether the message contains an image and present an image view controller if it does.

- (void)conversationViewController:(ATLConversationViewController *)viewController didSelectMessage:(LYRMessage *)message {
    LYRMessagePart *JPEGMessagePart = ATLMessagePartForMIMEType(message, ATLMIMETypeImageJPEG);
    if (JPEGMessagePart) {
        [self presentImageViewControllerWithMessage:message];
        return;
    }
    LYRMessagePart *PNGMessagePart = ATLMessagePartForMIMEType(message, ATLMIMETypeImagePNG);
    if (PNGMessagePart) {
        [self presentImageViewControllerWithMessage:message];
    }
}

In order to show the image, we first need to load the image from Layer. Create a method loadLowResImageForMessage: that loads the low resolution image for a message.

- (UIImage *)loadLowResImageForMessage:(LYRMessage *)message {
    LYRMessagePart *lowResImagePart = ATLMessagePartForMIMEType(message, ATLMIMETypeImageJPEGPreview);
    LYRMessagePart *imageInfoPart = ATLMessagePartForMIMEType(message, ATLMIMETypeImageSize);

    if (!lowResImagePart) {
        // Default back to image/jpeg MIMEType
        lowResImagePart = ATLMessagePartForMIMEType(message, ATLMIMETypeImageJPEG);
    }

    // Retrieve low-res image from message part
    if (!(lowResImagePart.transferStatus == LYRContentTransferReadyForDownload || lowResImagePart.transferStatus == LYRContentTransferDownloading)) {
        if (lowResImagePart.fileURL) {
            return [UIImage imageWithContentsOfFile:lowResImagePart.fileURL.path];
        } else {
            return [UIImage imageWithData:lowResImagePart.data];
        }
    }
    return nil;
}

To present the image in a new view controller, we will use the JTSImageViewController library to simplify our work. Include the following in your Podfile and do pod install.

pod 'JTSImageViewController', '~> 1.4'

If you are using some other library, modify the following method accordingly.

- (void)presentImageViewControllerWithMessage:(LYRMessage *)message {
    // Create image info
    JTSImageInfo *imageInfo = [[JTSImageInfo alloc] init];
    imageInfo.image = [self loadLowResImageForMessage:message];

    // Setup view controller
    JTSImageViewController *imageViewer = [[JTSImageViewController alloc] initWithImageInfo:imageInfo mode:JTSImageViewControllerMode_Image backgroundStyle:JTSImageViewControllerBackgroundOption_Scaled message:message];

    // Present the view controller.
    [imageViewer showFromViewController:self transition:JTSImageViewControllerTransition_FromOffscreen];
}

However, this just loads the low resolution image in the view controller. Let’s initialize the view controller with the low res image and load the high resolution image in the background and replace it when it’s available. Create a subclass of JTSImageViewController named PGConversationImageViewController and use that in the above method instead of JTSImageViewController.

//
//  PGConversationImageViewController.h
//

#import "JTSImageViewController.h"

@interface PGConversationImageViewController : JTSImageViewController <LYRProgressDelegate>
@property(nonatomic, strong) LYRMessage *message;

- (id)initWithImageInfo:(JTSImageInfo *)info mode:(enum JTSImageViewControllerMode)mode backgroundStyle:(enum JTSImageViewControllerBackgroundOptions)style message:(LYRMessage *)message;
@end


//
//  PGConversationImageViewController.m
//

#import <LayerKit/LYRMessagePart.h>
#import <Atlas/Utilities/ATLMessagingUtilities.h>
#import "PGConversationImageViewController.h"

@interface PGConversationImageViewController ()
@property(nonatomic, strong) UIImage *fullResImage;
- (void)updateInterfaceWithImage:(UIImage *)image;
@end

@implementation PGConversationImageViewController
- (void)viewDidAppear:(BOOL)animated {
    [super viewDidAppear:animated];
    [self downloadFullResImageIfNeeded];
}

- (id)initWithImageInfo:(JTSImageInfo *)info mode:(enum JTSImageViewControllerMode)mode backgroundStyle:(enum JTSImageViewControllerBackgroundOptions)style message:(LYRMessage *)message {
    PGConversationImageViewController *_self = [self initWithImageInfo:self.imageInfo mode:mode backgroundStyle:style];
    if (_self) {
        _self.message = message;
    }
    return _self;
}

#pragma mark - Layer Image Downloads

- (void)loadFullResImages {
    LYRMessagePart *fullResImagePart = ATLMessagePartForMIMEType(self.message, ATLMIMETypeImageJPEG);
    if (!fullResImagePart) {
        fullResImagePart = ATLMessagePartForMIMEType(self.message, ATLMIMETypeImagePNG);
    }

    // Retrieve hi-res image from message part
    if (!(fullResImagePart.transferStatus == LYRContentTransferReadyForDownload || fullResImagePart.transferStatus == LYRContentTransferDownloading)) {
        if (fullResImagePart.fileURL) {
            self.fullResImage = [UIImage imageWithContentsOfFile:fullResImagePart.fileURL.path];
        } else {
            self.fullResImage = [UIImage imageWithData:fullResImagePart.data];
        }
        [self updateInterfaceWithImage:self.fullResImage];
    }
}

- (void)downloadFullResImageIfNeeded {
    LYRMessagePart *fullResImagePart = ATLMessagePartForMIMEType(self.message, ATLMIMETypeImageJPEG);
    if (!fullResImagePart) {
        fullResImagePart = ATLMessagePartForMIMEType(self.message, ATLMIMETypeImagePNG);
    }

    // Download hi-res image from the network
    if (fullResImagePart && (fullResImagePart.transferStatus == LYRContentTransferReadyForDownload || fullResImagePart.transferStatus == LYRContentTransferDownloading)) {
        NSError *error;
        LYRProgress *downloadProgress = [fullResImagePart downloadContent:&error];
        if (!downloadProgress) {
            NSLog(@"Problem downloading full resolution photo - %@", error);
            return;
        }
        downloadProgress.delegate = self;
    } else {
        [self loadFullResImages];
    }
}

#pragma mark - LYRProgress Delegate Implementation

- (void)progressDidChange:(LYRProgress *)progress {
    // Queue UI updates onto the main thread, since LYRProgress performs
    // delegate callbacks from a background thread.
    dispatch_async(dispatch_get_main_queue(), ^{
        BOOL progressCompleted = progress.fractionCompleted == 1.0f;
        // After transfer completes, remove self for delegation.
        if (progressCompleted) {
            progress.delegate = nil;
            self.title = @"Image Downloaded";
            [self loadFullResImages];
        }
    });
}

@end

Here’s how it should look now:

Need help setting this up or just want to chat? Shoot me an email at pulkit110@gmail.com.

How to show relative time in Atlas Conversation List Controller (Layer Chat)

Sat, 25 Apr 2015 06:32:55 GMT

Although, Atlas is highly customizable, there are a few things that it doesn’t have an option to customize. This post explains how you can update the conversation list in Atlas to include relative time for the same day instead of absolute time. Here’s how it looks by default:

![](https://res.cloudinary.com/http-pulkitgoyal-in/image/upload/c_scale,h_465/AtlasConversationListAbsoluteTime_jgn3ft.png)

Digging into the code, you will find that the cells in converstaion list are built in ATLConversationTableViewCell. There it defaults to using an absolute time if the date the message was sent on was today. We can create a subclass of the cell and override the date formatting methods to return a relative time instead. Let’s use the DateTools library to handle conversion of NSDates to relative time strings. Add this to your Podfile and do a pod install

pod 'DateTools', '~> 1.5'

Now create a subclass of ATLConversationTableViewCell named PGConversationListTableViewCell.

//
//  PGConversationListTableViewCell.h
//
#import "ATLConversationTableViewCell.h"
@interface PGConversationListTableViewCell : ATLConversationTableViewCell <ATLConversationPresenting>
@end

//
//  PGConversationListTableViewCell.m
//
#import "PGConversationListTableViewCell.h"
#import "NSDate+DateTools.h"

@interface PGConversationListTableViewCell ()
- (NSString *)dateLabelForLastMessage:(LYRMessage *)lastMessage;
@end

@implementation PGConversationListTableViewCell

- (NSString *)dateLabelForLastMessage:(LYRMessage *)lastMessage {
    if (!lastMessage) return @"";
    if (!lastMessage.receivedAt) return @"";
    return lastMessage.receivedAt.timeAgoSinceNow;
}
@end

Now, all we need to do is make Atlas pick up this class instead of the default one when laying out the table view. We can do this when customizing our ATLConversationListViewController. If you created a subclass of the conversation list controller, do this somewhere in viewDidLoad. Otherwise, set the cell class on the ATLConversationListViewController instance when you set up the controller.

[self setCellClass:[PGConversationListTableViewCell class]];

Here’s how your conversation list should look now. Easy, right?

![](https://res.cloudinary.com/http-pulkitgoyal-in/image/upload/c_scale,h_465/AtlasConversationListRelativeTime_qerwvv.png)

Need help setting this up or just want to chat? Shoot me an email at pulkit110@gmail.com.

Implementing Chat/Messaging in iOS apps with Layer

Thu, 09 Apr 2015 18:49:27 GMT

Introduction

Layer is an open communications layer for the Internet, a platform that enables app creators everywhere to easily add native communications to their products.

Whether an app is centered around compelling content in sports, entertainment or news — or the focus is finding and buying a concert ticket or a t-shirt, booking a flight or hotel, or hailing a town car or taxi — enabling users to engage with one another in-app and in context is the key to a delightful user experience.

Layer is one of the easiest ways to integrate chat and messsaging features in your apps. With the recent launch of Atlas, it is even easier to integrate Layer in your app using its fully featured, high performance, 100% customizable UI kit.

Implementation

This tutorial explains how to quickly get started with Layer and Atlas to create a simple app to chat between a device and a simulator.

Configuration

Set up Layer

The first step is to connect the layer client. Add the following to your AppDelegate.m

static NSString *const kLayerAppID = @"LAYER-APP-ID"; // TODO Update layer app id here

// ...

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {

    // ...
    
    // Setup Layer
    _layerClient = [LYRClient clientWithAppID:[[NSUUID alloc] initWithUUIDString:kLayerAppID]];
    self.layerClient.autodownloadMIMETypes = [NSSet setWithObjects:ATLMIMETypeImageJPEGPreview, ATLMIMETypeTextPlain, nil];
    [self.layerClient connectWithCompletion:^(BOOL success, NSError *error) {
        if (!success) {
            NSLog(@"Failed to connect to Layer: %@", error);
        } else {
            // TODO This will usually be in a view controller after the user authenticates
            // For the purposes of this Quick Start project, let's authenticate as a user named 'Device/Simulator'.
    #if TARGET_IPHONE_SIMULATOR
            NSString *userIDString = @"Simulator";
    #else // TARGET_IPHONE_SIMULATOR
            NSString *userIDString = @"Device";
    #endif // TARGET_IPHONE_SIMULATOR

            // Once connected, authenticate user.
            // Check Authenticate step for authenticateLayerWithUserID source
            [self authenticateLayerWithUserID:userIDString completion:^(BOOL success, NSError *error) {
                if (!success) {
                    NSLog(@"Failed Authenticating Layer Client with error:%@", error);
                }
            }];
        }
    }];
}

Authenticate

Once connected, it is time to authenticate the LYRClient object. Layer authentication requires that a backend server generate an Identity Token on behalf of the client. For testing purposes, Layer provides a sample backend that takes care of this.

- (void)authenticateLayerWithUserID:(NSString *)userID completion:(void (^)(BOOL success, NSError * error))completion
{
    // Check to see if the layerClient is already authenticated.
    if (self.layerClient.authenticatedUserID) {
        // If the layerClient is authenticated with the requested userID, complete the authentication process.
        if ([self.layerClient.authenticatedUserID isEqualToString:userID]){
            NSLog(@"Layer Authenticated as User %@", self.layerClient.authenticatedUserID);
            if (completion) completion(YES, nil);
            return;
        } else {
            //If the authenticated userID is different, then deauthenticate the current client and re-authenticate with the new userID.
            [self.layerClient deauthenticateWithCompletion:^(BOOL success, NSError *error) {
                if (!error){
                    [self authenticationTokenWithUserId:userID completion:^(BOOL success, NSError *error) {
                        if (completion){
                            completion(success, error);
                        }
                    }];
                } else {
                    if (completion){
                        completion(NO, error);
                    }
                }
            }];
        }
    } else {
        // If the layerClient isn't already authenticated, then authenticate.
        [self authenticationTokenWithUserId:userID completion:^(BOOL success, NSError *error) {
            if (completion){
                completion(success, error);
            }
        }];
    }
}

- (void)authenticationTokenWithUserId:(NSString *)userID completion:(void (^)(BOOL success, NSError* error))completion{

    /*
     * 1. Request an authentication Nonce from Layer
     */
    [self.layerClient requestAuthenticationNonceWithCompletion:^(NSString *nonce, NSError *error) {
        if (!nonce) {
            if (completion) {
                completion(NO, error);
            }
            return;
        }

        /*
         * 2. Acquire identity Token from Layer Identity Service
         */
        [self requestIdentityTokenForUserID:userID appID:[self.layerClient.appID UUIDString] nonce:nonce completion:^(NSString *identityToken, NSError *error) {
            if (!identityToken) {
                if (completion) {
                    completion(NO, error);
                }
                return;
            }

            /*
             * 3. Submit identity token to Layer for validation
             */
            [self.layerClient authenticateWithIdentityToken:identityToken completion:^(NSString *authenticatedUserID, NSError *error) {
                if (authenticatedUserID) {
                    if (completion) {
                        completion(YES, nil);
                    }
                    NSLog(@"Layer Authenticated as User: %@", authenticatedUserID);
                } else {
                    completion(NO, error);
                }
            }];
        }];
    }];
}

The following code snippet connects to the sample Layer Identity Service, generates an Identity Token on behalf your application, and authenticates the LYRClient. Note that in production apps, you would need to update it to communicate with your backend and generate a token for Layer. Please check out the Layer Authentication Guide for setting up your backend for this.

- (void)requestIdentityTokenForUserID:(NSString *)userID appID:(NSString *)appID nonce:(NSString *)nonce completion:(void(^)(NSString *identityToken, NSError *error))completion
{
    NSParameterAssert(userID);
    NSParameterAssert(appID);
    NSParameterAssert(nonce);
    NSParameterAssert(completion);

    NSURL *identityTokenURL = [NSURL URLWithString:@"https://layer-identity-provider.herokuapp.com/identity_tokens"];
    NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:identityTokenURL];
    request.HTTPMethod = @"POST";
    [request setValue:@"application/json" forHTTPHeaderField:@"Content-Type"];
    [request setValue:@"application/json" forHTTPHeaderField:@"Accept"];

    NSDictionary *parameters = @{ @"app_id": appID, @"user_id": userID, @"nonce": nonce };
    NSData *requestBody = [NSJSONSerialization dataWithJSONObject:parameters options:0 error:nil];
    request.HTTPBody = requestBody;

    NSURLSessionConfiguration *sessionConfiguration = [NSURLSessionConfiguration ephemeralSessionConfiguration];
    NSURLSession *session = [NSURLSession sessionWithConfiguration:sessionConfiguration];
    [[session dataTaskWithRequest:request completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
        if (error) {
            completion(nil, error);
            return;
        }

        // Deserialize the response
        NSDictionary *responseObject = [NSJSONSerialization JSONObjectWithData:data options:0 error:nil];
        if(![responseObject valueForKey:@"error"])
        {
            NSString *identityToken = responseObject[@"identity_token"];
            completion(identityToken, nil);
        }
        else
        {
            NSString *domain = @"layer-identity-provider.herokuapp.com";
            NSInteger code = [responseObject[@"status"] integerValue];
            NSDictionary *userInfo =
            @{
               NSLocalizedDescriptionKey: @"Layer Identity Provider Returned an Error.",
               NSLocalizedRecoverySuggestionErrorKey: @"There may be a problem with your APPID."
            };

            NSError *error = [[NSError alloc] initWithDomain:domain code:code userInfo:userInfo];
            completion(nil, error);
        }

    }] resume];
}

Building the UI

Participant

A participant is a user that participates in a conversation. It contains information for Atlas to render the details about the user when displaying conversations. Create a class PGUser that implements the ATLParticipant and ATLAvatarItem protocols. For the simple example, it uses the participant identifier as all properties of the users, but you will need to modify it to fetch these properties from your API based on the participant identifier.

// PGUser.h
@interface PGUser : NSObject <ATLParticipant, ATLAvatarItem>

@property(nonatomic, readonly) NSString *firstName;
@property(nonatomic, readonly) NSString *lastName;
@property(nonatomic, readonly) NSString *fullName;
@property(nonatomic, readonly) NSString *participantIdentifier;
@property(nonatomic, readonly) UIImage *avatarImage;
@property(nonatomic, readonly) NSString *avatarInitials;

- (instancetype)initWithParticipantIdentifier:(NSString *)participantIdentifier;

+ (instancetype)userWithParticipantIdentifier:(NSString *)participantIdentifier;


@end

// PGUser.m
@implementation PGUser

- (instancetype)initWithParticipantIdentifier:(NSString *)participantIdentifier {
    self = [super init];
    if (self) {
        _participantIdentifier = participantIdentifier;
        _firstName = participantIdentifier;
        _lastName = participantIdentifier;
        _fullName = participantIdentifier;
        _avatarInitials = [participantIdentifier substringToIndex:1];
    }

    return self;
}

+ (instancetype)userWithParticipantIdentifier:(NSString *)participantIdentifier {
    return [[self alloc] initWithParticipantIdentifier:participantIdentifier];
}

@end

Show all user conversations

The user is now authenticated to Layer. Let’s grab and show all conversations the user is currently involved in. Create a subclass PGConversationListViewController of ATLConversationListViewController.

// PGConversationListViewController.h
@interface PGConversationListViewController : ATLConversationListViewController <ATLParticipantTableViewControllerDelegate>
- (void)selectConversation:(LYRConversation *)conversation;
+ (NSSet *)participants;
@end


// PGConversationListViewController.m
@implementation PGConversationListViewController
- (void)viewDidLoad {
    [super viewDidLoad];
	
    // Set data source and delegate
    self.dataSource = self;
    self.delegate = self;

	// Add a create chat button
    self.navigationItem.rightBarButtonItem = [[UIBarButtonItem alloc] initWithBarButtonSystemItem:UIBarButtonSystemItemAdd target:self action:@selector(createNewChat:)];
}

#pragma mark - Conversation List Data Source

- (NSString *)conversationListViewController:(ATLConversationListViewController *)conversationListViewController titleForConversation:(LYRConversation *)conversation {
    NSMutableSet *participantIdentifiers = [conversation.participants mutableCopy];
    [participantIdentifiers minusSet:[NSSet setWithObject:self.layerClient.authenticatedUserID]];

    if (participantIdentifiers.count == 0) return @"Personal Conversation";
    NSMutableSet *participants = [[self participantsForIdentifiers:participantIdentifiers] mutableCopy];
    if (participants.count == 0) return @"No Matching Participants";
    if (participants.count == 1) return [[participants allObjects][0] fullName];

    NSMutableArray *firstNames = [NSMutableArray new];
    [participants enumerateObjectsUsingBlock:^(id obj, BOOL *stop) {
        id <ATLParticipant> participant = obj;
        if (participant.firstName) {
            // Put the last message sender's name first
            if ([conversation.lastMessage.sentByUserID isEqualToString:participant.participantIdentifier]) {
                [firstNames insertObject:participant.firstName atIndex:0];
            } else {
                [firstNames addObject:participant.firstName];
            }
        }
    }];
    NSString *firstNamesString = [firstNames componentsJoinedByString:@", "];
    return firstNamesString;
}

#pragma mark - Conversation List Delegate

- (void)conversationListViewController:(ATLConversationListViewController *)conversationListViewController didSelectConversation:(LYRConversation *)conversation {
    [self presentConversationControllerForConversation:conversation];
}

#pragma mark - Participant Delegate

- (void)participantTableViewController:(ATLParticipantTableViewController *)participantTableViewController didSelectParticipant:(id <ATLParticipant>)participant {
    [self.navigationController popViewControllerAnimated:NO];

    // Create a new conversation
    NSError *error = nil;
    LYRConversation *conversation = [self.layerClient newConversationWithParticipants:[NSSet setWithArray:@[self.layerClient.authenticatedUserID, participant.participantIdentifier]] options:nil error:&error];
    if (!conversation) {
        NSLog(@"New Conversation creation failed: %@", error);
    } else {
        [self presentConversationControllerForConversation:conversation];
    }
}

- (void)participantTableViewController:(ATLParticipantTableViewController *)participantTableViewController didSearchWithString:(NSString *)searchText completion:(void (^)(NSSet *filteredParticipants))completion {
}

#pragma mark - Helpers

- (void)presentConversationControllerForConversation:(LYRConversation *)conversation {
    ATLConversationViewController *conversationViewController = [PGConversationViewController conversationViewControllerWithLayerClient:self.layerClient];
    conversationViewController.conversation = conversation;
    [self.navigationController pushViewController:conversationViewController animated:YES];
}

- (NSSet *)participantsForIdentifiers:(NSSet *)identifiers {
    NSMutableSet *participants = [[NSMutableSet alloc] initWithCapacity:identifiers.count];
    for (NSString *identifier in identifiers) {
        [participants addObject:[PGUser userWithParticipantIdentifier:identifier]];
    }
    return participants;
}

#pragma mark - User Interaction

- (void)createNewChat:(id)sender {
    ATLParticipantTableViewController *participantTableViewController = [ATLParticipantTableViewController participantTableViewControllerWithParticipants:[PGConversationListViewController participants] sortType:ATLParticipantPickerSortTypeFirstName];
    participantTableViewController.delegate = self;
    [self.navigationController pushViewController:participantTableViewController animated:YES];
}

#pragma mark - Static

+ (NSSet *)participants {

    __block NSSet *participants;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        participants = [NSSet setWithArray:@[[PGUser userWithParticipantIdentifier:@"Device"], [PGUser userWithParticipantIdentifier:@"Simulator"]]];
    });
    return participants;
}

@end

Just present this controller to open all chats the user is currently involved in. Let’s present this from our AppDelegate for now. Add this to the end of application:didFinishLaunchingWithOptions: (Note: It assumes that the initial view controller in the storyboard is a UINavigationController)

UINavigationController *navigationController = (UINavigationController *) self.window.rootViewController; 
self.conversationListViewController = [LCConversationListViewController conversationListViewControllerWithLayerClient:self.layerClient];
[navigationController pushViewController:self.conversationListViewController animated:NO];

Show all messages in a conversation

Now that we have all the conversations the user is involved in, let’s see how we can show all messages inside a single conversation. Create a subclass PGConversationViewController of ATLConversationViewController.

// PGConversationViewController.h
@interface PGConversationViewController : ATLConversationViewController <ATLConversationViewControllerDataSource, ATLConversationViewControllerDelegate>

@end

// PGConversationViewController.m
@implementation PGConversationViewController

- (void)viewDidLoad {
    [super viewDidLoad];
    self.delegate = self;
    self.dataSource = self;
}

#pragma mark - Conversation Data Source

- (id <ATLParticipant>)conversationViewController:(ATLConversationViewController *)conversationViewController participantForIdentifier:(NSString *)participantIdentifier {
    // TODO Return the user corresponding to this participant identifier
    return [PGUser userWithParticipantIdentifier:participantIdentifier];
}

- (NSAttributedString *)conversationViewController:(ATLConversationViewController *)conversationViewController attributedStringForDisplayOfDate:(NSDate *)date {
    return [[NSAttributedString alloc] initWithString:[NSDateFormatter localizedStringFromDate:date dateStyle:NSDateFormatterShortStyle timeStyle:NSDateFormatterShortStyle]];
}

- (NSAttributedString *)conversationViewController:(ATLConversationViewController *)conversationViewController attributedStringForDisplayOfRecipientStatus:(NSDictionary *)recipientStatus {
    NSMutableDictionary *mutableRecipientStatus = [recipientStatus mutableCopy];
    if ([mutableRecipientStatus valueForKey:self.layerClient.authenticatedUserID]) {
        [mutableRecipientStatus removeObjectForKey:self.layerClient.authenticatedUserID];
    }

    NSString *statusString = [NSString new];
    if (mutableRecipientStatus.count > 1) {
        __block NSUInteger readCount = 0;
        __block BOOL delivered;
        __block BOOL sent;
        [mutableRecipientStatus enumerateKeysAndObjectsUsingBlock:^(NSString *userID, NSNumber *statusNumber, BOOL *stop) {
            LYRRecipientStatus status = (LYRRecipientStatus) statusNumber.integerValue;
            switch (status) {
                case LYRRecipientStatusInvalid:
                    break;
                case LYRRecipientStatusSent:
                    sent = YES;
                    break;
                case LYRRecipientStatusDelivered:
                    delivered = YES;
                    break;
                case LYRRecipientStatusRead:
                    NSLog(@"Read");
                    readCount += 1;
                    break;
            }
        }];
        if (readCount) {
            NSString *participantString = readCount > 1 ? @"Participants" : @"Participant";
            statusString = [NSString stringWithFormat:@"Read by %lu %@", (unsigned long) readCount, participantString];
        } else if (delivered) {
            statusString = @"Delivered";
        } else if (sent) {
            statusString = @"Sent";
        }
    } else {
        __block NSString *blockStatusString = [NSString new];
        [mutableRecipientStatus enumerateKeysAndObjectsUsingBlock:^(NSString *userID, NSNumber *statusNumber, BOOL *stop) {
            if ([userID isEqualToString:self.layerClient.authenticatedUserID]) return;
            LYRRecipientStatus status = (LYRRecipientStatus) statusNumber.integerValue;
            switch (status) {
                case LYRRecipientStatusInvalid:
                    blockStatusString = @"Not Sent";
                case LYRRecipientStatusSent:
                    blockStatusString = @"Sent";
                case LYRRecipientStatusDelivered:
                    blockStatusString = @"Delivered";
                    break;
                case LYRRecipientStatusRead:
                    blockStatusString = @"Read";
                    break;
            }
        }];
        statusString = blockStatusString;
    }
    return [[NSAttributedString alloc] initWithString:statusString attributes:@{NSFontAttributeName : [UIFont boldSystemFontOfSize:11]}];

}

@end

That’s all you need to integrate a chat in your application. Need help setting this up or just want to chat? Shoot me an email at pulkit110@gmail.com.

Organizing ChromeApp Development with Grunt

Sun, 22 Mar 2015 14:45:59 GMT

Chrome Apps deliver an experience as capable as a native app, but as safe as a web page. Just like web apps, Chrome Apps are written in HTML5, JavaScript, and CSS. But Chrome Apps look and behave like native apps, and they have native-like capabilities that are much more powerful than those available to web apps.

The Grunt ecosystem is huge and it’s growing every day. With literally hundreds of plugins to choose from, you can use Grunt to automate just about anything with a minimum of effort. Here is what we used to automate our tasks for developing, building and publishing the chrome app all with grunt.

Develop

grunt-wiredep: Inject Bower packages into your source code with Grunt. Just run this task after you install a new dependency with bower and it will insert it in areas marked inside your html file.

<!-- bower:js -->
<!-- endbower -->

Here’s what our configuration for the task looks like:

wiredep: {
    app: {
        src: ['<%= config.app %>/index.html'],
        ignorePath: '<%= config.app %>/'
    }
};

grunt-usemin: Replaces references from non-optimized scripts, stylesheets and other assets to their optimized version within a set of HTML files (or any templates/views). Just add all your scripts in html inside a build block and get another file with generated scripts/html optimized for deployment.

<!-- build:js js/app.js -->
<script src="js/app.js"></script>
<script src="js/controllers/thing-controller.js"></script>
<script src="js/models/thing-model.js"></script>
<script src="js/views/thing-view.js"></script>
<!-- endbuild -->

Here’s how our usemin configuration looks like:

usemin: {
    options: {
        assetsDirs: ['<%= config.dist %>', '<%= config.dist %>/images']
    },
    html: ['<%= config.dist %>/{,*/}*.html'],
    css: ['<%= config.dist %>/styles/{,*/}*.css']
},
useminPrepare: {
    options: {
        dest: '<%= config.dist %>'
    },
    html: ['<%= config.processed %>/index.html', '<%= config.processed %>/player.html']
}

grunt-env, grunt-preprocess and grunt-contrib-copy: Use grunt-env to specify different environment variables for different targets. It works great when coupled with grunt-preprocess which can trim out the code based on the environment. For example, to have different app names for staging and production, you could do the following in `manifest.json:
```
{
"name": "",
...
}
```
Here’s the configuration:

// Apply environment variables
env: {
    staging: {
        APP_NAME: 'App Dev'
    },
    prod: {
        APP_NAME: 'App'
    }
},
// Copy files from `app` directory to `processed` directory
copy: {
    preprocess: {
        expand: true,
        cwd: '<%= config.app %>',
        dest: '<%= config.processed %>',
        src: ['**']
    }
},
// Preprocess all js, html and json files inside the `processed` directory
preprocess: {
    inline: {
        src: ['<%= config.processed %>/js/{,*/}*.js', '<%= config.processed %>/*.html', '<%= config.processed %>/*.json'],
        options: {
            inline: true
        }
    }
}

grunt-contrib-jshint: Make sure code styles are up to par and there are no obvious mistakes.

// Make sure code styles are up to par and there are no obvious mistakes
jshint: {
    options: {
        jshintrc: '.jshintrc',
        reporter: require('jshint-stylish')
    },
    all: [
        'Gruntfile.js',
        '<%= config.processed %>/js/{,*/}*.js',
        '!<%= config.processed %>/js/vendor/*',
        'test/spec/{,*/}*.js'
    ]
}

grunt-contrib-watch: Run predefined tasks whenever watched file patterns are added, changed or deleted.

watch: {
// Run the wiredep task whenever bower.json changes
bower: {
    files: ['bower.json'],
    tasks: ['wiredep', 'copy:preprocess', 'preprocess']
},
// Run preprocess and jshint whenever any js file changes
js: {
    files: ['<%= config.app %>/js/**/*.js'],
    tasks: ['copy:preprocess', 'preprocess', 'jshint'],
    options: {
        livereload: 45729
    }
},
// Run preprocess whenever any css file changes
styles: {
    files: ['<%= config.app %>/styles/{,*/}*.css'],
    tasks: ['copy:preprocess', 'preprocess'],
    options: {
        livereload: 45729
    }
},
// Reload the app on any significant changes
livereload: {
    tasks: ['copy:preprocess', 'preprocess'],
    options: {
        livereload: 45729
    },
    files: [
        '.tmp/styles/{,*/}*.css',
        '<%= config.app %>/*.html',
        '<%= config.app %>/images/{,*/}*.{png,jpg,jpeg,gif,webp,svg}',
        '<%= config.app %>/manifest.json',
        '<%= config.app %>/_locales/{,*/}*.json'
    ]
}
}

Build and Deploy

grunt-chrome-manifest: Grunt task for Chrome manifest.json

// Merge event page, update build number, exclude the debug script
chromeManifest: {
    dist: {
        options: {
        	// Don't increase the build number automatically
            buildnumber: false,
            background: {
                target: 'js/background.js',
                // Exclude files used for development
                exclude: [
                    'js/chromereload.js'
                ]
            }
        },
        src: '<%= config.processed %>',
        dest: '<%= config.dist %>'
    }
}

grunt-bump: Automatically bump version numbers, create tags, commit releases etc.

// Increments the version
bump: {
    options: {
	    // Update the version in package.json, manifest.json and commit the changes
        files: ['package.json', '<%= config.app %>/manifest.json'],
        updateConfigs: [],
        commit: true,
        commitMessage: 'Release v%VERSION%',
        commitFiles: ['package.json', '<%= config.app %>/manifest.json'],
        createTag: false,
        push: false,
        pushTo: 'upstream',
        gitDescribeOptions: '--tags --always --abbrev=1 --dirty=-d'
    }
}

grunt-contrib-compress and grunt-webstore-upload: Compress the packaged app and automate uploading process of the new versions of Chrome Extension or App to Chrome Webstore.

// Copmress files inside processed directory and create a new package ready for upload
compress: function (grunt) {
    return {
        dist: {
            options: {
                archive: function () {
                    var manifest = grunt.file.readJSON(grunt.template.process('<%= config.processed %>') + '/manifest.json');
                    return grunt.template.process('<%= config.package %>') + '/App-' + manifest.version + '.zip';
                }
            },
            files: [
                {
                    expand: true,
                    cwd: 'dist/',
                    src: ['**'],
                    dest: ''
                }
            ]
        }
    }
},
// Uploads latest build from package to production/staging
webstore: {
    options: {
        publish: true,
        client_id: "XXXX",
        client_secret: "XXXX"
    },
    "staging": {
        options: {
            appID: "XXXX",
            zip: "<%= config.package %>" // Uploads the most recent zip file from this directory
        }
    }
}

Putting everything together: Lets put all the tasks together into more manageable defaults for our development and deployment.

// Configurable paths
var config = {
    app: 'app',
    dist: 'dist',
    processed: 'processed',
    package: 'package',
    tasks: grunt.cli.tasks
};

// Loads task options from `tasks/options/` and loads tasks defined in `package.json`
// Read more here: https://www.thomasboyt.com/2013/09/01/maintainable-grunt.html
require('load-grunt-config')(grunt, {
    configPath: path.join(process.cwd(), 'tasks/options'),
    init: true,
    config: {
        pkg: grunt.file.readJSON('package.json'),
        manifest: 'manifest',
        config: config
    }
});

var target = grunt.option('target') || 'dev';

// `grunt debug [--target=dev|test|staging|prod]` - Default, run Chrome App on Chrome app container
grunt.registerTask('debug', 'Task for development of the app. Enables live reload', [
    'clean',
    'env:' + target,
    'copy:preprocess',
    'preprocess',
    'watch'
]);

// Creates a production/staging build and archives to zip file
// grunt build [--target=dev|test|staging|prod]
grunt.registerTask('build', 'Creates a production/staging build and archives to zip file', [
    'env:' + target,
    'clean:dist',
    'copy:preprocess',
    'preprocess',
    'chromeManifest:dist',
    'useminPrepare',
    'concat',
    'cssmin',
    'uglify',
    'copy:dist',
    'copy:styles',
    'usemin',
    'compress'
]);

// Publish the application on the web store. Automatically increases the build version, uploads the build and commits the updated files.
// grunt publish [--release=major|minor|patch] [--target=staging|prod] [--setversion=x.x.x]
grunt.registerTask('publish', 'Publish the application on the web store', [
    'bump-only:' + release,
    'build',
    'webstore_upload:' + target,
    'bump-commit'
]);

By the way, if you enjoy movies and tv shows, check out our MovieTabs extension for Chrome to discover new movies and tv shows with every tab.

Things you should know about Auto Layout on iOS

Thu, 11 Dec 2014 11:02:17 GMT

After spending some time with developers working with iOS, I have noticed that a lot of developers out there struggle with Auto Layout. I have found that there are a few simple rules that you need to understand to make working with auto layout a delight. Most of the following will depend on how your views are layed out, but this is usually what I do when laying out views in the storyboard.

Introduction

Auto Layout is a system that lets you lay out your app’s user interface by creating a mathematical description of the relationships between the elements. You define these relationships in terms of constraints either on individual elements, or between sets of elements. Using Auto Layout, you can create a dynamic and versatile interface that responds appropriately to changes in screen size, device orientation, and localization.

Auto Layout is on by default for new projects that you create in XCode. The typical workflow for designing with Auto Layout is to create, reposition, resize and customize your views and then set up the constraints for the views so that they maintain their position on all devices. With the introduction of

Until now, if your designs were reasonably complex, you had to write a lot of code to support such adaptive layouts. Not only does Auto Layout makes it easy to support different screen sizes in your apps, as a bonus it also makes internationalization almost trivial.

The basics

You can specify constraints to pin the spacing between views, to pin the dimension of the views or to define the alignment of the views.

Try to keep Update Frames to None unless you are sure that your auto layout constraints are complete for the view you are positioning. When you update frames, XCode tries to position the views by satisfying the constraints and if the constraints are not complete, most of the times, it will end up resizing your view to 0 width/height. If you do leave Update Frames on and XCode repositions view to someplace you didn’t expect it to, do a Cmd+Z to bring the views back to their old position without removing the constraints you added.
When in doubt, check the color of the small arrow next to the view controller you are designing. Red means there are some errors in the auto layout constraints that you added. Yellow means that some frames are incorrectly positioned with respect to the constraints.

![](https://res.cloudinary.com/http-pulkitgoyal-in/image/upload/c_scale,w_400/v1418058060/Screen_Shot_2014-12-08_at_18_00_15_z1s9kt.png)

Clicking on the arrow will take you to a list of errors/warnings. You can get suggestions on how to fix it by clicking on the small circle on the right.

![](https://res.cloudinary.com/http-pulkitgoyal-in/image/upload/c_scale,w_400/v1418058275/Screen_Shot_2014-12-08_at_18_03_10_vgz4ku.png)

Deleting a constraint will not reposition your views but the layout might go into “warning” mode, indicating issues you need to resolve to make auto-layout work.
When you move things around, your constraints don’t change; only view frame in IB changes. When frame in IB is actually different from calculated with layout constraints frame, IB will indicate the runtime frame of that element with yellow dotted line and the layout will be in “warning” mode.
Once you have added the constraints for a view, you can update the frames of all its subviews pressing the small triangle button in the bottom after selecting the view you want to update.

![](https://res.cloudinary.com/http-pulkitgoyal-in/image/upload/c_scale,w_400/v1418058476/Screen_Shot_2014-12-08_at_18_05_56_e8dfps.png)

If you cannot select a constraint from the Storyboard, select the view and open the Size Inspector to select the constraint from there.

![](https://res.cloudinary.com/http-pulkitgoyal-in/image/upload/c_scale,w_400/v1418058830/Screen_Shot_2014-12-08_at_18_13_31_kq1gpe.png)

Animations should now be dealt in terms of auto layout constraints. This means that you should not manually touch the frame when animating. Change the constraints and call layoutIfNeeded inside the animation block. To build the constraints in code, I recommend using Masonry rather than the overly verbose defaults.

Some Tips

In each dimension (vertical and horizontal), each view’s position and size are defined by three values: leading space, size and trailing space. The leading and trailing spaces can be defined either in terms of a view’s superview or in relation to a sibling in the view hierarchy. Generally speaking, your layout constraints must fix two of these values so that the third one can be calculated. As a result, a standard view needs at least two constraints in each dimension for an unambiguous layout.
Try to position views based on the leading and trailing spaces (or the alignment constraints) instead of pinning their heights/widths. Again, this will depend on how your views are layed out, but this is something that holds true in general for me.
Auto Layout for scroll views works differently than normal views. There is an in depth explaination about making auto layout work with scroll view in the technical note from Apple. The gist of it is 1) you will need more constraints (usually in form of pinned width/height for vertical/horizontal scrolling respectively) than with normal views because scroll view also needs to compute its content size, and 2) there should be chain of constraints that link one edge of the scroll view to its opposite edge (e.g. in the vertical dimension, there should be a top spacing constraint between your topmost subview and the scroll view, a bottom spacing constraint between the botttommost subview and the scrollview, and a chain of constraints that link the bottom edge of the topmost subview to the top edge of the bottommost subview).

I know that it sounds complex, but its not. Go try it out in XCode and I am sure you will never switch back once you understand how it works.

Need some help or just want to chat? Shoot me an email at pulkit110@gmail.com.

Pluralization on iOS and OS X

Tue, 25 Nov 2014 20:11:39 GMT

As of iOS 7 and Mac OS X 10.9 Mavericks, Foundation has the ability to specify localized strings according to pluralization and grammar rules. So no need to write code like this to deal with strings whose conjugations change based on a dynamic value anymore.

if (count == 1) {
  return NSLocalizedString(@"1 Person", nil);
} else {
  return [NSString stringWithFormat:NSLocalizedString(@"%d People", nil), count];
}

There is a new file format with .stringsdict suffix introduced. [NSBundle localizedStringForKey:value:table:] now accesses two files: .stringsdict and .strings. It queries the .stringsdict file first, then, .strings file.

The .stringsdict file contains the original key (e.g. @“%d files are selected”) with the value as an arbitrary property list (mostly dictionary or string). When the value is a dictionary, it must contain a key for the localized format string value and its format specifier configuration dictionaries. Here’s an example:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>%d files are selected</key>
    <dict>
        <key>NSStringLocalizedFormatKey</key>
        <string>%#@num_files@ selected</string>
        <key>num_files</key>
        <dict>
            <key>NSStringFormatSpecTypeKey</key>
            <string>NSStringPluralRuleType</string>
            <key>NSStringFormatValueTypeKey</key>
            <string>d</string>
            <key>zero</key>
            <string>No file is</string>
            <key>one</key>
            <string>A file is</string>
            <key>other</key>
            <string>%d files are</string>
        </dict>
    </dict>
</dict>
</plist>

A configuration key between two @s is used to query an item in the external format configuration dictionary.

Here, the num_files is the key defined in the string using the enhanced # format specifier and will be replaced depending on the passed value when using localizedStringWithFormat:. Its a bit difficult to understand at first looking at the xml file. Let us see how this plist will look like in json:

"%d files are selected": {
    "NSStringLocalizedFormatKey": "%#@num_files@ selected", // The string to format
    "num_files": {
        "NSStringFormatSpecTypeKey": "NSStringPluralRuleType", // Its a plural rule
        "NSStringFormatValueTypeKey": "d", // We expect an integer argument
        "zero": "No file is", // String to use in place of `num_files` when value is zero
        "one": "A file is",
        "other": "%d files are"
    }
}

How to use it: Just create a localizable.stringsdict file and add it to your XCode project and use it in code:

label.text = [NSString localizedStringWithFormat:NSLocalizedString(@"%d files are selected", @"%d files are selected"), numFilesSelected];

Note: It might not work on the simulator straight away, so you might want to try it on a device before you start pulling your hairs out!

iOS Parse SDK Overview

Fri, 14 Nov 2014 21:05:20 GMT

This post is intended to be a very brief introduction to the Parse SDK for iOS. It just touches the most basic points when setting up your app with Parse. Refer to the official guide for an in depth view of what’s possible with Parse.

Storing data on Parse is built around the PFObject. Each PFObject contains key-value pairs of JSON-compatible data. This data is schemaless, which means that you don’t need to specify ahead of time what keys exist on each PFObject. I highly recommend creating classes for your models derived from PFObject to avoid inconsistencies with manually handling key value pairs. To do so, just subclass PFObject and implement the PFSubclassing prototype. Define all columns on the backend as @dynamic properties in the class like so:

@interface QBQuote : PFObject <PFSubclassing>

// Properties on backend
@property(nonatomic, strong) NSString *text; // String column on the backend
@property(nonatomic, strong) QBImage *image; // Pointer<Image> on the backend
@property(nonatomic, strong) NSArray *tags; // Array<Tag> on the backend

// Computed properties (no need for `@dynamic` in implementation)
@property(nonatomic, readonly) BOOL favorite;

@end

Of course, you will need classes like QBTag and QBImage registered for their respective tables on the backend.

@implementation QBQuote
@dynamic text, image, tags;

#pragma mark - Parse

+ (void)load {
    // Required to register the sub class with Parse
    [self registerSubclass];
}

+ (NSString *)parseClassName {
    // Name of class on the backend
    return @"Quote";
}

+ (PFQuery *)query {
    PFQuery *query = [super query];

    // Include some child keys to avoid making another requests for them in the future
    [query includeKey:@"image"];
    [query includeKey:@"tags"];

    // Load from the cache if network requests fail
    query.cachePolicy = kPFCachePolicyNetworkElseCache;

    return query;
}

@end

This ia all you need to start making requests for data from the backend. I like to keep all my backend logic in the class, so I have a few more methods here for loading data from the backend. Here’s a sample implementation of loading the data using PromiseKit (Check out this post for a quick overview). Here is how you would retreive objects (using Parse-PromiseKit):

#pragma mark - Backend

+ (PMKPromise *)loadWithFilterBlock:(void (^)(PFQuery *query))filterBlock page:(int)page {
    PFQuery *query = [QBQuote query];
    query.limit = kQBQuotePerPageLimit;
    query.skip = page * kQBQuotePerPageLimit;

    // Apply additional filters
    if (filterBlock) {
        filterBlock(query);
    }

    return [query promiseFindObjects];
}

+ (PMKPromise *)countWithFilterBlock:(void (^)(PFQuery *query))filterBlock {
    PFQuery *query = [QBQuote query];

    // Apply additional filters
    if (filterBlock) {
        filterBlock(query);
    }

    return [query promiseCountObjects];
}

Loading items is now just a simple call like this:

[QBQuote loadWithFilterBlock:self.filterblock page:self.numberOfQuotes / kQBQuotePerPageLimit].then(^(NSArray *quotes) { ... }];

Need to save objects? Just update the properties, and call saveEventually and it will take care of handling the network outages and other faults. An important thing to keep in mind when creating/updating objects is the permission for the objects. Use the ACL property on PFObject to control who has access to the object. For example, to allow write access to an object by only the current user:

quote.ACL = [PFACL ACLWithUser:[PFUser currentUser]];
[quote.ACL setPublicReadAccess:YES];

Scanning Barcodes and QR Codes with AVFoundation on iOS7+

Sun, 12 Oct 2014 08:37:31 GMT

With the release of iOS 7, Apple included the ability to detect barcodes through its AVFoundation framework. It is really easy to add barcode scanning to your app without using any additional libraries.

We just need to create a capture session so we can trigger the camera when needed. We set the input for the capture session to our capture device (the device camera). The output is an AVCaptureMetadataOutput object which takes care of reading the metadata from camera input. This is the core step of reading the bar codes. To see what the camera captures, we can add a preview layer to the controller’s view and attach it to the camera. Finally, to start scanning, start the session and you are done. The delegate methods will be called when a barcode is detected.

// Find the input for default capture device
AVCaptureDevice *captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:captureDevice error:&error];
if (!input) {
    NSLog(@"%@", [error localizedDescription]);
    return NO;
}

// Create a capture session and add input
self.captureSession = [[AVCaptureSession alloc] init];
[self.captureSession addInput:input];

// Add the session output
AVCaptureMetadataOutput *captureMetadataOutput = [[AVCaptureMetadataOutput alloc] init];
[self.captureSession addOutput:captureMetadataOutput];

// Configure the metadata output
dispatch_queue_t dispatchQueue;
dispatchQueue = dispatch_queue_create("barcodeReaderQueue", NULL);
[captureMetadataOutput setMetadataObjectsDelegate:self queue:dispatchQueue];
[captureMetadataOutput setMetadataObjectTypes:@[AVMetadataObjectTypeEAN13Code, AVMetadataObjectTypeCode128Code, AVMetadataObjectTypeCode93Code, AVMetadataObjectTypeCode39Code]];

// Add a preview layer to see camera preview
self.videoPreviewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:self.captureSession];
[self.videoPreviewLayer setVideoGravity:AVLayerVideoGravityResizeAspectFill];
[self.videoPreviewLayer setFrame:self.previewView.layer.bounds];
[self.previewView.layer addSublayer:self.videoPreviewLayer];

// Start running the capture session
[self.captureSession startRunning];

The delegate method captureOutput:didOutputMetadataObjects:fromConnection: is called everytime a metadata object is processed. From here, we can extract the string value of the metadata object to get the barcode string.

#pragma mark - Capture Metadata Output Objects Delegate
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputMetadataObjects:(NSArray *)metadataObjects fromConnection:(AVCaptureConnection *)connection {
    if (metadataObjects != nil && [metadataObjects count] > 0) {
        AVMetadataMachineReadableCodeObject *metadataObj = [metadataObjects objectAtIndex:0];
        NSString *barcodeString = metadataObj.stringValue;
        NSLog(@"Read barcode: %@", barcodeString);
    }
}

Configuring MMDrawer for Side Drawer Navigation on iOS - Part II

Fri, 26 Sep 2014 17:20:55 GMT

In an earlier post, I presented an example of setting up the side drawer navigation pattern for iOS using Storyboards and posted an example project based on the described set up. This post expands on that set up to add some basic configuration to the side drawer.

In the example app, the side drawer is first initialized when the DRAWER_SEGUE is triggered. All the configuration for MMDrawerController should be done in the prepareForSegue: for this segue. Here’s the basic configuration that we have for the drawer controller in PGViewController.m where we get the MMDrawerController instance from the segue and set a centerViewController and a leftDrawerViewController.

- (void)prepareForSegue:(UIStoryboardSegue *)segue sender:(id)sender {
    if ([segue.identifier isEqualToString:@"DRAWER_SEGUE"]) {
        MMDrawerController *destinationViewController = (MMDrawerController *) segue.destinationViewController;

        // Instantitate and set the center view controller.
        UIViewController *centerViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"FIRST_TOP_VIEW_CONTROLLER"];
        [destinationViewController setCenterViewController:centerViewController];

        // Instantiate and set the left drawer controller.
        UIViewController *leftDrawerViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"SIDE_DRAWER_CONTROLLER"];
        [destinationViewController setLeftDrawerViewController:leftDrawerViewController];
    }
}

Let’s add a few more configuration options to the drawer to enable gesture support for opening and closing the drawer controller, a maximum drawer width and a parallax animation.

// Set maximum drawer width
destinationViewController.maximumLeftDrawerWidth = 254;
// Use all gestures for closing the drawer
destinationViewController.closeDrawerGestureModeMask = MMCloseDrawerGestureModeAll;
// Open the drawer only when panning the center view from the edge
destinationViewController.openDrawerGestureModeMask = MMOpenDrawerGestureModeBezelPanningCenterView;
// Set a parallax animation for opening drawer
[destinationViewController setDrawerVisualStateBlock:[MMDrawerVisualState parallaxVisualStateBlockWithParallaxFactor:5]];

MMDrawerController also comes with a built in menu button. Let’s see how we can add it to a controller.

- (void)setupLeftMenuButton {
    MMDrawerBarButtonItem * leftDrawerButton = [[MMDrawerBarButtonItem alloc] initWithTarget:self action:@selector(leftDrawerButtonPress:)];
    [self.navigationItem setLeftBarButtonItem:leftDrawerButton];
}

- (void)leftDrawerButtonPress:(id)leftDrawerButtonPress {
    [self.mm_drawerController toggleDrawerSide:MMDrawerSideLeft animated:YES completion:nil];
}

Starting from iOS7, the views are drawn beneath the status bar. If you want to control the style of the status bar when the side drawer is opened, all you need is to override the preferredStatusBarStyle in the controller that is set as the [left|right]DrawerViewController (PGSideDrawerController in the example project).

#pragma mark - Status Bar
-(UIStatusBarStyle)preferredStatusBarStyle{
    return UIStatusBarStyleLightContent;
}

Multiplayer 2048 Game for iOS - Part II

Sun, 14 Sep 2014 14:10:41 GMT

We launched the 2048 Game on iOS in April 2014. I have already written some details about why we did it and the initial launch in a post here. We weren’t looking to build just another clone, there were tons of them already and we were aware of it. We wanted to push out our idea of how it can be tunrned into a multiplayer game. Here are some details about the multiplayer version and how it turned out for us (TLDR: If you just want the gist of it, it didn’t go very well or at least as well as we had planned).

The rules of multiplayer 2048

Two players share the same board and get points only for the tiles that they merge. In every turn, in addition to scoring more points, the player had to think about how to stop the other one from scoring in his next turn. Sort of like chess, sort of like the Letterpress app.

The multiplayer version

We knew straightaway that we want to use GameCenter, Apple’s social gaming network to build the multiplayer mode. The turn based APIs matched our needs perfectly. Here are the major tasks that a turn based match needs to handle

Allow the local player to join a match.
Allow the local player to see the list of existing matches.
Allow the local player to view the state of a match in progress.
Allow the player to take a turn in the match.
When a player leaves a match, set the player’s match outcome.
When all the players have a match outcome set, end the match.
Handle invitations and other match events.
Expand your game to include exchanges.

I wouldn’t bore you with too much technical details in this post. If you are curious, head over to the github and check out the code for our open source version.

The launch

So, all done with the turn based stuff and tests, we were heading to launch with great expectations. We thought that even if people have half of our excitement for the multiplayer version, it will still go great. So, we submit the multiplayer version to apple and after a week of wait its finally approved for App Store. We had been playing against each other the whole week waiting to finally get the chance to play against other players when the app launches. I started creating new matches against random guys and I played some intense matches with people I never knew. It was fun!

So, after a few matches, we hop on to Mixpanel to see how many guys are actually trying out the multiplayer version and there was the surprise for us. Not many! Most of the guys just kept on playing the single player version and never bothered to check out the multiplayer mode. We were disappointed, we had put so much effort into this. Well, we had our moments, those who actually did try the multiplayer version enjoyed a lot.

What we learnt

We try to extract positives from every new idea that we try out. It was fun building multiplayer 2048. It was amazing to see people enjoying the game. But it was heartbreaking to see so few of them! There were a few things that we might have done differently to increase our chances.

We didn’t put a limit on the amount of time a player could take to play his move. This allowed players to take a move and then not play for hours (or even days) which ruined the experience. We could have been omre aggressive in forcing the players to take the turn as soon as possible (e.g. by deducting time based on how long they take to copmlete the move or by giving an extra move to the second player)
We were a bit late to the party. The game had already crossed its peak and there were plenty of clones out there which meant that it was really hard to find our version on the app store.
This one might sound like an excuse, but the Android version of 2048 launched its own multiplayer while we were waiting for Apple to approve ours. Although, the multiplayer version in the Android app was so different from ours (I wouldn’t even call it multiplayer, 2 players played completely independent of each other on their own boards for a limited amount of time and then compared their scores when the time is over). It took some interest away from our multiplayer version and people didn’t care to notice thinking that it would be the same as the Android counterpart.

Anyway, thanks for reading. It was fun. And if you like what you see, do try out the game here.

Building the 2048 game for iOS

Wed, 10 Sep 2014 18:19:43 GMT

We launched the 2048 Game on iOS in April 2014. We started developing the iOS version of the game (based on the initial web based version from Gabriel Cirullli) after the game was already well pass its peak.

Why we did it?

There were enough clones already, right? Why did we build another one? Well, we had this cool idea of placing two players against each other (as in the Letterpress Game) on the same board, playing in turns, competing to merge as many tiles as possible. The rules were simple, two players share the same board and get points only for the tiles that they merge. In every turn, in addition to scoring more points, the player had to think about how to stop the other one from scoring in his next turn. We enjoyed this format so much that we used to play this on the traditional web version by manually noting scores and taking turns.

The first version

With the ultimate goal of building the multiplayer community of players competing against each other, we started working on the first version. We wanted to keep it simple, but we wanted to keep it native, not some webview based clone that just bluntly opens the website in the browser. It was the first game that we were going to develop, so we had no idea where to start. The board was just tiles, so we used the UICollectionView to put tiles in, added some game logic and it was done. Not yet!

The default animations in the UICollectionView assumed that all the items came from just one group which means that moving an item out of a row meant that all other items in next rows change their positions. We had to use fake cells to perform the animations between items and completely disable the default UICollectionView animations. Not too small, but nothing copmlicated either. Take a look.

We even added a simple tilt control by using CoreMotion that allows you to move tiles by tilting the phone. Check out the deviceMotionDidUpdate: method to see the details.

The launch

We launched the first version without the multiplayer option to the AppStore in hopes of getting enough users for the app so that when the multiplayer version comes out, we would have a community of players already there to try out the game and players don’t have to wait a lot to find another player to play with. Was this move successful? Sort of. We got close to two thousand players in a few weeks. I know that this wasn’t great, but it wasn’t too bad either considering there were already tons of clones doing the same thing already.

More details about the multiplayer version here.

Resizing High Resolution Images on iOS Without Memory Issues

Sun, 31 Aug 2014 11:11:50 GMT

If you have ever worked with user generated images in an iOS app, you would know that it is very easy to quickly spend all the memory you have which would then result in your app being terminated by the OS. In order to support high resolution images in an app, I searched around for ways to resize images on device. The easiest and the most common way is to use CoreGraphics to draw the image to a smaller rect and then convert it back to a UIImage. The problem with this approach is that it isn’t gaining you anything because for drawing the image, you are decoding it anyway which requires reading the full sized image and might crash the app.

To correctly resize the image without decoding it first is to drop down a level and use the ImageIO framework. The first thing you need with the ImageIO framework is to create an image source from an image (the image can be an existing UIImage, path to a file or even an ALAsset). Let’s see how we can resize an image when we know the local path to the image. If you have the image in some other form, all you need to change is this step (see this for details).

The next step is to use the CGImageSourceCreateThumbnailAtIndex method to actually generate the thumbnail image from this source. We need to pass some options here to configure the thumbnail generation (e.g. the max pixel size). kCGImageSourceCreateThumbnailWithTransform option automatically rotates the image to the correct orientation and kCGImageSourceCreateThumbnailFromImageAlways asks the OS to create the thumbnail even if it is already present. This option is important as without it, the OS might use an existing thumbnail which can be much smaller than what you asked for.

- (void)resizeImageAtPath:(NSString *)imagePath {
	// Create the image source
    CGImageSourceRef src = CGImageSourceCreateWithURL((__bridge CFURLRef) [NSURL fileURLWithPath:imagePath], NULL);
    // Create thumbnail options
    CFDictionaryRef options = (__bridge CFDictionaryRef) @{
            (id) kCGImageSourceCreateThumbnailWithTransform : @YES,
            (id) kCGImageSourceCreateThumbnailFromImageAlways : @YES,
            (id) kCGImageSourceThumbnailMaxPixelSize : @(640)
    };
    // Generate the thumbnail
    CGImageRef thumbnail = CGImageSourceCreateThumbnailAtIndex(src, 0, options);
    CFRelease(src);
    // Write the thumbnail at path
    CGImageWriteToFile(thumbnail, imagePath);
}

Now that we have the CGImageRef for the small image, let’s see how we can save the image back to the path. We will use the CGImageDestinationAddImage function to write the image.

void CGImageWriteToFile(CGImageRef image, NSString *path) {
    CFURLRef url = (__bridge CFURLRef) [NSURL fileURLWithPath:path];
    CGImageDestinationRef destination = CGImageDestinationCreateWithURL(url, kUTTypePNG, 1, NULL);
    CGImageDestinationAddImage(destination, image, nil);

    if (!CGImageDestinationFinalize(destination)) {
        NSLog(@"Failed to write image to %@", path);
    }
}

This is all you need to load larger images in form of something that you can safely understand. We even tried this with a 6019×6019 image from NASA without any memory problems.

Using Grunt to Auto Upload Files to Android

Tue, 19 Aug 2014 18:33:50 GMT

Grunt is a JavaScript task runner. It is being used heavily for web development to automate repititive tasks like minification, compilation, unit testing, linting, etc. After you’ve configured it, a task runner can do most of that mundane work for you—and your team—with basically zero effort.

For a recent project, I had to repeatedly upload some files to my Android device whenever some changes were made on the code base. Being familiar with adb, it was my immediate choice to upload files. Just a simple command adp push and your files are on the device in no time. Since I was already working with Grunt on the project with a watch task configured to identify when some files change, it was really simple to add another task to run the command to upload files.

First you would need the grunt-shell task to execute shell commands. Then, just put in the configuration for shell task and you are ready to go.

shell : {
  sync_android: {
    command: "<%= adbPath %> push 'path/to/local/directory' '<%= pathOnDevice %>'"
  }
}

The variables adbPath and pathOnDevice are defined in grunt config as follows:

config: {
  adbPath: '/Applications/Android\\ Studio.app/sdk/platform-tools/adb',
  pathOnDevice: grunt.option('android_path') || '<default/path/on/device>'
}

Now, just run the task from command line: grunt shell:sync_android [--android_path=] or wire with watch to auto upload files.

Retrieving Videos from YouTube Channel in Objective-C API v3

Tue, 05 Aug 2014 15:58:01 GMT

The YouTube Data API (v3) lets you incorporate YouTube functionality into your own application. You can use the API to fetch search results and to retrieve, insert, update, and delete resources like videos or playlists.

This blog post gives a quick overview of sending queries using the Obj-C Library. Lets see how to get all videos for a channel with known id. First, we will send a list channel query with our channel identifier to load the channel info. Then we execute a list playlist items query to find all videos in the playlists. Here’s some code:

- (void)loadChannels {
    GTLQueryYouTube *channelsQuery = [GTLQueryYouTube queryForChannelsListWithPart:@"id,snippet,brandingSettings,contentDetails,invideoPromotion,statistics,status,topicDetails"];
    channelsQuery.identifier = @"<CHANNEL_IDENTIFIER>";

    [self.youTubeService executeQuery:channelsQuery completionHandler:^(GTLServiceTicket *ticket, GTLYouTubeChannelListResponse *channelListResponse, NSError *error) {
        if (error == nil) {
            if (channelListResponse.items.count > 0) {
                GTLYouTubeChannel *channel = channelListResponse.items.firstObject;
                NSLog(@"Got youtube channel - %@", channel);
                [self loadPlaylistsForChannel:channel];
            } else {
                NSLog(@"Cannot find channels with id %@", channelsQuery.identifier);
            }

        } else {
            NSLog(@"Cannot get youtube channels - %@", error);
        }
    }];
}

- (void)loadPlaylistsForChannel:(GTLYouTubeChannel *)channel {
    NSString *playlistId = channel.contentDetails.relatedPlaylists.uploads;
    if (playlistId) {
        GTLQueryYouTube *playlistsQuery = [GTLQueryYouTube queryForPlaylistItemsListWithPart:@"id,snippet,contentDetails,status"];
        playlistsQuery.playlistId = playlistId;
        [self.youTubeService executeQuery:playlistsQuery completionHandler:^(GTLServiceTicket *ticket, GTLYouTubePlaylistItemListResponse *playlistListResponse, NSError *error) {
            if (error == nil) {
                if (playlistListResponse.items.count > 0) {
                    self.playlistItems = playlistListResponse.items;
                    [self.swipeView reloadData];
                    for (GTLYouTubePlaylistItem *playlistItem in playlistListResponse.items) {
                        NSLog(@"Got youtube playlist items - %@", playlistItem);
                    }
                    [self loadVideoForPlaylistItem:playlistListResponse.items.firstObject];
                } else {
                    NSLog(@"Cannot find playlists items with playlist id %@", playlistId);
                }
            } else {
                NSLog(@"Cannot get youtube playlist items - %@", error);
            }
        }];
    } else {
        NSLog(@"Playlist 'uploads' not found for channel.");
    }
}

There are several queries available from the API. For the complete list, head over to the API Reference.

iOS CoreData Overview

Sat, 21 Jun 2014 18:06:35 GMT

This post is intended to be a very brief introduction to Core Data. It just touches the most important points when setting up a core data stack.

The schema for the database is defined in my-project.xcdatamodeld. This is where you tell CoreData what entities to expect and what attributes and relationships go in each entity. The .h and .m files for the entities can be generated automatically from the schema using XCode (some details here). Just open .xcdatamodelfile, click on the entity, and go to File->New->File. Select the Core Data\NSManagedObject subclass, and click Next, and then click Create again on the following view.

Its better to not play with these because they will be overwritten whenever you make changes in the schema and re-generate the files. To add custom methods and properties to these entities, I usually prefer creating categories on the entity (MyEntity+Extensions.h,m).

I use MagicalRecord to perform CoreData requests. We could do it using the native CoreData APIs, but MagicalRecord makes it a lot easier and intuitive. Fetching from the database is done with the MR_ methods on the entity. e.g. [MyEntity MR_findAll] will return an array of all news items in the database. Filtering can be done with predicates (e.g. this fetches all news for the given date).

Introducing Multiplayer mode for 2048

Sat, 03 May 2014 09:27:49 GMT

Today, we’re introducing a completely new experience for enjoying 2048 game on your iPhone. With the 2nd version of our 2048 game, you will now see that there is an option to play against other players in a multiplayer match. We have put a lot of thought and time in trying to come up with a good gameplay experience for this multiplayer mode.

You can choose to start a new multiplayer match with a randomly chosen opponent or invite your friends to play with. The game is played on a single board shared between both the players who take turns to play. The score of a player is computed by the number on the tiles that the player merges when playing his turn. The game ends when there are no moves remaining or someone reaches the 2048 tile. At this point, the player with the highest score wins the game.

So what are you waiting for, get the app from the AppStore now and start competing with your friends. We promise, it will be fun!

This is cross posted from The Shyahi Blog. If you like what you see, I encourage you to sign up for the RSS feed or follow me on Twitter.

Pull to Refresh with iOS 7 style cover image behind Navigation Bar

Tue, 11 Mar 2014 06:21:24 GMT

A very common interface design that we have started seeing with iOS7 apps is having a big cover photo on the top of table views which extends all the way up to the status bar. Here’s an example. However, by default, iOS adds an inset to the scroll views so that the content starts from below the top layout guide. Its easy to put everything back to top, though. All we need is to set the contentInset of tableview to UIEdgeInsetsMake(0, 0, 0, 0).

Lets try to put a refresh control to our tableview now. I like to use a UIViewController instead of a UITableViewController to have more control over the views. Here’s a hack to make the refresh control work with UIViewController.

// Add pull to refresh control.
UITableViewController *tableViewController = [[UITableViewController alloc] init];
tableViewController.tableView = self.tableView;
self.refreshControl = [[UIRefreshControl alloc] init];
[self.refreshControl addTarget:self action:@selector(doRefresh) forControlEvents:UIControlEventValueChanged];
tableViewController.refreshControl = self.refreshControl;

Now try pulling the tableview down. So, we have the refresh control, but it is hidden behind the navbar. The pull to refresh control starts from the content inset for the tableview and we already set it to 0 to have our cover image nicely fit behind the navbar. Time for another quick hack to get this to work. We need to offset the content in the header view by 64 pixels (the length of the top layout guide) and reset the top content inset for tableview to 64 pixels so that the cover image starts from behind the navigation bar and the pull to refresh from below.

if (SYSTEM_VERSION_GREATER_THAN_OR_EQUAL_TO(@"7.0")) {
        self.tableView.contentInset = UIEdgeInsetsMake(self.topLayoutGuide.length, 0, 0, 0); // This is the default and might not be necessary
        self.headerContainerViewConstraint.constant = -self.topLayoutGuide.length;
        self.tableView.tableHeaderView.frame = CGRectMake(self.tableView.tableHeaderView.frame.origin.x, self.tableView.tableHeaderView.frame.origin.y, self.tableView.tableHeaderView.frame.size.width, self.tableView.tableHeaderView.frame.size.height - self.topLayoutGuide.length);
}

Handling authenticated activities after logout [Android]

Sun, 23 Feb 2014 08:04:54 GMT

Handling authenticated users is a very common scenario in a lot of applications. In such a scenario, you might have some activities which should be accessible to only authenticated users and there might be serveral of these activities open in the stack. How do you handle log out from an arbitrary stage in the app when you might have several activities that depend on authentication pushed on the back stack?

One way is to send the user to a “Login” activity with its flag set to FLAG_ACTIVITY_CLEAR_TOP on log out which will clear all activities in the back stack upto the login activity. But this wouldn’t work if you can have saved sessions between app restarts where there might not even be an existing login activity when the user logs out.

Another option would be to check the log in status in onCreate and finish() the activity if the user is not logged in. Seems hacky, doesn’t it?

Lets think about what we want to do here. We want multiple activities to be able to be notified as soon as the user logs out. This is the task for a BroadcastReceiver. Basically we need to brodcast a logout message to all our Activities that need to stay under a logged-in status. We can use the sendBroadcast and install a BroadcastReceiver in all our Actvities.

// After logging out
Intent broadcastIntent = new Intent();
broadcastIntent.setAction("com.package.ACTION_LOGOUT");
sendBroadcast(broadcastIntent);

And for all the activities that require authentication, register a receiver in onCreate. Make sure to also call LocalBroadcastManager.getInstance(this).unregisterReceiver() in onDestroy() so that we don’t receive any more broadcasts.

protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);

    IntentFilter intentFilter = new IntentFilter();
    intentFilter.addAction("com.package.ACTION_LOGOUT");
    LocalBroadcastManager.getInstance(this).registerReceiver(new BroadcastReceiver() {
        @Override
        public void onReceive(Context context, Intent intent) {
            Log.d("onReceive","Logout in progress");
            //At this point we should start the login activity and finish this one.
            finish();
        }
    }, intentFilter);
}

Add params to Ember Data's RESTAdapter AJAX Requests

Sun, 26 Jan 2014 21:55:17 GMT

Ember Data is a persistence layer for Ember.Js. Ember Data is a library that integrates tightly with Ember.js to make it easy to retrieve records from a server, cache them for performance, save updates back to the server, and create new records on the client.

Ember Data makes it really simple to communicate with a RESTful JSON API by automatically creating appropriate urls for REST API calls and serializing/desserialzing parameters as required. However, for certain tasks, it is important to be able to customize certain things that Ember Data automates for you. For example, you might want to add some parameters to all the requests that you send through Ember Data in order to authenticate the user. As with everything else in Ember, you could do it with very little extra code. Just override the ajax method in RESTAdapter to add your params and you are done.

App.RESTAdapter = DS.RESTAdapter.extend({
  // Scope all ajax calls.
  ajax: function(url, type, hash) {
    if (Ember.isEmpty(hash)) hash = {};
    if (Ember.isEmpty(hash.data)) hash.data = {};
    hash.data.access_token = App.accessToken; // Add an access token param.
    return this._super(url, type, hash);
  }
});

Building RESTful iOS apps using RestKit

Tue, 24 Dec 2013 20:02:10 GMT

RestKit is an Objective-C framework for iOS that aims to make interacting with RESTful web services simple, fast and fun. It combines a clean, simple HTTP request/response API with a powerful object mapping system that reduces the amount of code you need to write to get stuff done.

This post is not about getting started with RestKit. There are several excellent posts already available that talk about it. This is about how you can structure your apps to better leverage the RestKit functionality and keep the code maintainable and easy to follow.

RKRouter is something that isn’t described in great detail in the existing guides. It is also something that is very powerful and is great to put together all the routes for your REST requests. An RKRouter instance is responsible for generating NSURL objects with a given base URL and a route set. It is used to centralize the knowledge about the URL’s that are used by the application.

URL’s can be generated by the router in three ways:

By name. Named routes link a symbolic name with a path and an HTTP request method.
By object. Routes can be defined by class and HTTP request method. When a URL is requested from the router for an object, the router will identify the most appropriate route for the object and instantiate an NSURL with the route’s path pattern and interpolate it against the object.
By object relationship. Routes can be defined for relationships to other objects. When a URL is requested from the router for a relationship, the router will retrieve the appropriate route for the relationship from the route set and interpolate the route’s path pattern against the source object.

Let see how we can organise our app to leverage the capabilities of RKRouter. We create an interface for each of our model that we expect in the response from our RESTful API.

@interface GHUser

@property (nonatomic, retain) NSString* name;
@property (nonatomic, retain) NSString* login;
@property (nonatomic, retain) NSString* email;
@property (nonatomic, retain) NSArray* repositories

+ (RKObjectMapping *) responseMapping;

@end

@implementation GHUser

+ (NSDictionary*)elementToPropertyMappings {
  return @{
    @"name": @"name",
    @"login": @"login",
    @"email": @"email",
  };
}

+ (RKObjectMapping *) responseMapping {
  // Create an object mapping.
  RKObjectMapping *mapping = [RKObjectMapping mappingForClass:[GHUser class]];
  [mapping addAttributeMappingsFromDictionary:[GHUser elementToPropertyMappings]];

  // Add some relation mappings (if any.)
  [mapping addPropertyMapping:[RKRelationshipMapping relationshipMappingFromKeyPath:@"repositories" toKeyPath:@"repositories" withMapping:[GMRepository responseMapping]]];

  return mapping;
}

@end

We want to separate the object mapping for each of the objects into their own class rather than putting them at one place where we declare the routes in order to make this manageable. We do this by creating a static method called responseMapping on each of our model which we can then use to get the mapping that we need. If you also need to send some objects as request body, you can declare a method like requestMapping to create a mapping that can be used in a RKRequestDescriptor. The simplest implementation of the requestMapping could be return [[GHUser responseMapping] inverseMapping];.

With this done, we can start creating some routes. Let’s have a interface called RestClient that will manage all the routes and also a shared object manager that holds everything.

// -- RestClient.h -- //

// Declare constants.
extern NSString *const kRestClientListUsersRouteName;

// Interface declaration.
@interface RestClient : NSObject

    + (instancetype)sharedClient;

@end

// -- RestClient.m -- //

// Define constants.
NSString *const kRestClientListUsersRouteName = @"LIST_USERS_ROUTE";

// Interface implementation.
@implementation RestClient

static RestClient *sharedClient = nil;

- (id)init {
    self = [super init];
    if (self) {
        [self initializeObjectManagers];

        // Set shared manager if nil
        if (nil == sharedClient) {
            [RestClient setSharedClient:self];
        }
    }

    return self;
}

- (void)initializeObjectManagers {
    // Uncomment below to enable rest kit logging.
    // RKLogConfigureByName("RestKit/Network", RKLogLevelTrace);

    // Configure the object manager.
    RKObjectManager *objectManager = [RKObjectManager managerWithBaseURL:[NSURL URLWithString:@"https://github.com/"]];
    objectManager.requestSerializationMIMEType = RKMIMETypeJSON;
    [RKObjectManager setSharedManager:objectManager];

    // -- Declare routes -- //

    // List users route. We create a named route here.
    RKObjectMapping *userMapping = [GHUser responseMapping];
    RKResponseDescriptor *listUsersResponseDescriptor = [RKResponseDescriptor responseDescriptorWithMapping:userMapping method:RKRequestMethodGET pathPattern:@"api/v2/users" keyPath:@"users" statusCodes:RKStatusCodeIndexSetForClass(RKStatusCodeClassSuccessful)];
    [objectManager addResponseDescriptor:listUsersResponseDescriptor];
    [objectManager.router.routeSet addRoute:[RKRoute routeWithName:kRestClientListUsersRouteName pathPattern:@"api/v2/users" method:RKRequestMethodGET]];

    // Get user by name route. We create a class route here.
    RKResponseDescriptor *userResponseDescriptor = [RKResponseDescriptor responseDescriptorWithMapping:userMapping method:RKRequestMethodGET pathPattern:@"api/v2/users/:name" keyPath:@"user" statusCodes:RKStatusCodeIndexSetForClass(RKStatusCodeClassSuccessful)];
    [objectManager addResponseDescriptor:userResponseDescriptor];
    [objectManager.router.routeSet addRoute:[RKRoute routeWithClass:[GMuser class] pathPattern:@"api/v2/users/:name" method:RKRequestMethodGET]];

    // Create user route. We want to send the user in the body of POST request.
    RKObjectMapping *userRequestMapping = [GMUser requestMapping];
    RKRequestDescriptor *userRequestDescriptor = [RKRequestDescriptor requestDescriptorWithMapping:userRequestMapping objectClass:[GMUser class] rootKeyPath:@"user" method:RKRequestMethodPOST];
    [objectManager addRequestDescriptor:userRequestDescriptor];
    RKResponseDescriptor *createUserResponseDescriptor = [RKResponseDescriptor responseDescriptorWithMapping:userMapping method:RKRequestMethodPOST pathPattern:@"api/v2/users" keyPath:@"user" statusCodes:RKStatusCodeIndexSetForClass(RKStatusCodeClassSuccessful)];
    [objectManager addResponseDescriptor:createUserResponseDescriptor];
    [objectManager.router.routeSet addRoute:[RKRoute routeWithClass:[GMUser class] pathPattern:@"api/v2/users" method:RKRequestMethodPOST]];
}

#pragma mark - Shared Client
+ (void)setSharedClient:(RestClient *)client {
    sharedClient = client;
}

+ (instancetype)sharedClient {
    return sharedClient;
}
@end

As you see, we add up all the routes for our REST requests in our rest client. If we want to separate things further, we might create a new static method in the GHUser interface that builds up the routes specific to the user. Anyway, after this, all we need to do is to initialise the rest client to store all the routes in. You can do this in you AppDelegate like so: [[RestClient alloc] init];.

Now, executing the requests is a simple call to the appropriate RKObjectManager method. e.g. To list all the users,

// Get list of users.
[[RKObjectManager sharedManager] getObjectsAtPathForRouteNamed:kRestClientListUsersRouteName
                                                            object:nil
                                                        parameters:parameters
                                                           success:^(RKObjectRequestOperation *operation, RKMappingResult *mappingResult) {
                                                               // Do something with mappingResult.array.
                                                           }
                                                           failure:^(RKObjectRequestOperation *operation, NSError *error) {
                                                               // Do something.
                                                           }];

// Get a user by user name.
[[RKObjectManager sharedManager] getObject:[GHUser userWithName:@"pulkit110"] path:nil parameters:nil
                                   success:^(RKObjectRequestOperation *operation, RKMappingResult *mappingResult) {
                                       // Do something with [mappingResult firstObject]
                                   }
                                   failure:^(RKObjectRequestOperation *operation, NSError *error) {
                                       // Do something
                                   }];

// Create a new user.
[[RKObjectManager sharedManager] postObject:user path:nil parameters:nil
                                    success:^(RKObjectRequestOperation *operation, RKMappingResult *mappingResult) {

                                    }
                                    failure:^(RKObjectRequestOperation *operation, NSError *error) {

                                    }

How to Implement Side Drawer Navigation for iOS

Mon, 23 Dec 2013 16:11:15 GMT

The slide-out design pattern lets developers add permanent navigation to their apps without taking up valuable screen real estate. The user can choose to reveal the navigation at any time, while still seeing their current context.

Its really easy to add one in your apps with wonderful libraries like MMDrawerController around. But most of the examples around just show an overly simplified view without any hints on how to structure your code for such a navigation pattern. This post tries to introduce an intuitive and easy to maintain structure for integrating MMDrawerController in an app with multiple contexts using Storyboard. Lets say, we want to have a root controller that does something and then we navigate to the drawer controller. The DrawerController is an instance of MMDrawerController and wraps a navigation drawer, and a center view controller. We will create everything in the Storyboard and use [self.storyboard instantiateViewControllerWithIdentifier:identifier] to instantiate the view controllers properly from there into our drawer controller’s leftDrawerViewController and centerViewController. Here’s how our Storyboard looks like:

Notice that we assign a storyboard identifier in the Identity Inspector so that we can instantiate the view controller from the Storyboard. Let’s assign identifier FIRST_TOP_VIEW_CONTROLLER to the navigation controller of FirstViewController so that we can use it as a center view controller. Now, all we need to do is to perform the segue to Drawer View Controller and instantiate the left and center controllers in the prepareForSegue: and set the controllers to the drawer controller like so:

-(void) prepareForSegue:(UIStoryboardSegue *)segue sender:(id)sender {
  if ([segue.identifier isEqualToString:@"DRAWER_SEGUE"]) {
    MMDrawerController *destinationViewController = (MMDrawerController *)segue.destinationViewController;

    // Instantitate and set the center view controller.
    UIViewController *centerViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"FIRST_TOP_VIEW_CONTROLLER"];
    [destinationViewController setCenterViewController: centerViewController];

    // Instantiate and set the left drawer controller.
    UIViewController *leftDrawerViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"SIDE_DRAWER_CONTROLLER"];
    [destinationViewController setLeftDrawerViewController: leftDrawerViewController];
  }
}

Now, we have a basic setup where we can segue to a navigation controller and instantiate the drawer and center controllers from the Storyboard. Now, how do we handle a user click on the the side drawer items to navigate to another controller in the center? Its simple! Just instantiate another view controller (like [self.storyboard instantiateViewControllerWithIdentifier:@"SECOND_TOP_VIEW_CONTROLLER"]) and set it as the center view controller for the drawer controller. You can access the drawer controller form any of your view controller using self.mm_drawerController.

[self.mm_drawerController setCenterViewController:[self.storyboard instantiateViewControllerWithIdentifier:@"SECOND_TOP_VIEW_CONTROLLER"] withCloseAnimation:YES completion:nil];

Just add more outlets from the side navigation drawer, instantiate new center controllers and set to the drawer controller. Its that simple to add a fully functional navigation drawer in your app. Now, you can keep on adding more controllers to each of you individual section just like you would in a normal app by creating more segues.

Here’s an implementation using this approach.

If you need a developer for your project, feel free to contact me at pulkit110@gmail.com

Uploading Objects to Amazon S3 Using Pre-Signed URLs

Fri, 25 Oct 2013 19:16:03 GMT

Amazon S3 is storage for the Internet. But you don't always want to give away the security credentials to anyone who wants to see/upload something on your bucket. This is where pre-signed urls come in. Anyone with valid security credentials can create a pre-signed URL. However, in order to successfully access an object, the pre-signed URL must be created by someone who has permission to perform the operation that the pre-signed URL is based upon. A pre-signed URL gives you access to the object identified in the URL.

I rencently came across the need to upoad an object to Amazon S3 using a pre signed url from an iOS app. There are some examples for Java, .NET and Ruby already available from Amazon. However, I couldn't find something similar for iOS. Here's an example on how you can upload an image to S3 using an presigned url from an iOS app.

-(void) uploadImage:(UIImage *)image atUrl:(NSString *)url {

  // Upload the image.
  NSData *imageData = UIImageJPEGRepresentation(image, 0.7);
  AFHTTPClient *client = [AFHTTPClient clientWithBaseURL:@"BASE_URL"]; // Initialize the client with some base url.

  // Create URL request.
  NSURL *requestURL = [NSURL URLWithString:[url stringByDecodingURLFormat]];
  NSMutableURLRequest *request = [[NSMutableURLRequest alloc] init];
  [request setHTTPMethod:@"PUT"];
  [request setValue:@"image/jpeg" forHTTPHeaderField:@"Content-Type"];
  [request setHTTPBody:imageData];
  [request setValue:[NSString stringWithFormat:@"%d", [imageData length]] forHTTPHeaderField:@"Content-Length"];
  [request setValue:@"public-read" forHTTPHeaderField:@"x-amz-acl"];
  [request setValue:@"iPhone-OS/6.0 fr_FR NE" forHTTPHeaderField:@"User-Agent"];
  [request setURL:requestURL];

  // Create the request operation.
  AFHTTPRequestOperation *requestOperation = [client HTTPRequestOperationWithRequest:request
                                                                             success:^(AFHTTPRequestOperation *operation, id responseObject) {
                                                                                 // Do something after successful upload.
                                                                             }
                                                                             failure:^(AFHTTPRequestOperation *operation, NSError *error) {
                                                                                 // Upload failed.
                                                                             }];

  [requestOperation setUploadProgressBlock:^(NSUInteger bytesWritten, long long int totalBytesWritten, long long int totalBytesExpectedToWrite) {
      float progress = (float) totalBytesWritten / (float) totalBytesExpectedToWrite;
      // Update progress on UI.
  }];

  // Begin uploading the file.
  [client enqueueHTTPRequestOperation:requestOperation];
}

A Review of Android remote image loading libraries

Tue, 09 Jul 2013 21:16:27 GMT

Loading remote images from an url into ImageViews in Android is one of the features that I have used quite extensively for almost all the Android apps that I have worked on. This isn’t really trivial as loading remote images requires creating an asynchronous task to download the image from url and then load it into the image view. There are several things that need to be taken care of when working with remote images because you can never know how large the image will be and loading a very big image into memory will cause the application to run out of memory very quicky.

While working on a project that relied heavily on external images, I tried out several of available libraries (UrlImageViewHelper, UniversalImageLoader and Picasso) to come up with the one that performs the best.

UrlImageViewLoader is a really light weight library and it works out of the box. All you need is to import the project into Eclipse, link it as a library project and start loading images into ImageViews with UrlImageViewHelper.setUrlDrawable(imageView, "https://example.com/image.png");. It performs callback methods, caching and works with ListAdapters. But this is where the fun stops. There are not a lot of configuration options but is really easy to quickly get started with.

Picasso is equally simple to use and performs advanced features like image cropping, resizing and transformations. It makes it very easy to resize the images if you already know the size of the image view you are loading the images into and the performance starts showing when you use resizing while loading a lot of images in a list view. This was something that I was using and Picasso immediately had an edge over UrlImageViewLoader for my requirements. The fact that it was developed by Jake Wharton (the developer of ActionBarSherlock) made it that much more appealing. But there were a lot of quirks and resizing didn’t seem to work always. It is still in very active development with a few commits at least every few days, but I would still wait for it to be more stable before trying it out again.

The final library in my review is UniversalImageLoader which is available as a jar file that can just be dropped into the project libraries. It is considerably heavy as compared with the other two libraries in the review, but has a lot of configuration options that allow controlling just about anything with the remote image loading. One of the most important things about it is the integrated resizing of images based on a maximum size (maybe not as efficient as Picasso, but definitely stable). The library has several caching strategies for the images as well as several display options. However, it does require a bit more configuration on the part of the developer. Here are the configuration options that I ended up using with UniversalImageLoader in my Application class.

// Create global configuration and initialize ImageLoader
DisplayImageOptions defaultOptions = new DisplayImageOptions.Builder().cacheInMemory().cacheOnDisc()
    .bitmapConfig(Bitmap.Config.RGB_565).imageScaleType(ImageScaleType.EXACTLY)
    .showStubImage(R.drawable.image_placeholder).showImageForEmptyUri(R.drawable.image_placeholder).build();
ImageLoaderConfiguration config = new ImageLoaderConfiguration.Builder(getApplicationContext())
    .defaultDisplayImageOptions(defaultOptions).build();
ImageLoader.getInstance().init(config);

Useful iOS resources

Sun, 05 May 2013 16:41:43 GMT

Its been quite some time since I began native iOS development and here's a post with my advice about some of the most useful resources for iOS developers.

CocoaPods: The first on my list is a wonderful dependency management library for Objective-C Projects (like Apache Maven if you are coming from the Java world). With CocoaPods setup for the project, all you need to do to use other libraries is to write them in the [cci]Podfile[/cci] and do [cci]pod install[/cci]. All you need to do now is include the files and write code.
CocoaLumberjack: CoocaLumberjack is logging system for iOS apps. I have found it to be one of the most useful frameworks out there, its fast, its easy to configure and its been actively developed. Bonus points for integrating it with XCodeColors to see coloured logs in your console.
AFNetworking: This is the one library that you will see in all such posts talking about resources for iOS developers. Why? Well, its really well designed, maintained and has a huge community of developers who have been using and contributing to it. If you want to handle network requests in your apps, you will not regret using this.
RestKit: This makes interacting with Rest services really easy and has a powerful object mapping system. RestKit's primary goal is to allow the developer to think more in terms of their application's data model and worry less about the details of sending requests, parsing responses, and building representations of remote resources.
MBProgressHUD: Again, one of the must use libraries. Provides beautiful HUDs and is really simple to integrate in the app.
Subtle Patterns: Subtle Patterns is a high quality (free) resource for tilable texture patterns. These patterns look really great on the iPhone and most of the patterns are usually optimized for retina display as well.
SDWebImage: Provides [cci]UIImageView[/cci] category for easily loading remote images from url. Really useful if you want to network images in your app.
TPKeyboardAvoiding: Provides UIViewController subclass that makes it easy to scroll your views to prevent them being hidden by the on screen keyboard.
GPUImage: This is more of a domain specific library, but is a must use if you find yourself trying to play with images, live camera feed or videos. It uses GPU where possible to perform image processing operations and makes it really easy to write custom OpenGL ES filters.
TestFlight and Diawi: These are two services that are very useful when you want to have beta testers for your app or want to let others on the team give a packaged version of the app that they can run on their devices. I usually prefer Diawi (for no specific reason though).
Cocoa Controls: A great resource to explore and search custom UI controls for iOS and Mac. I always search for available controls here before trying to make one. More often than not, I find something open sourced that I can just use/extend as per my needs.

Discussion on Reddit and Hacker News.

Creating video thumbnails using the iOS SDK

Fri, 12 Apr 2013 21:58:19 GMT

For a project that I have been working on recently, I came across the need for creating thumbnail images for videos in iOS application. The iOS SDK makes it really simple to create the thumbnail images through the AVAssetImageGenerator. AVAssetImageGenerator uses the default enabled video track(s) to generate images.

Generating a single image in isolation can require the decoding of a large number of video frames with complex interdependencies. If you require a series of images, you can achieve far greater efficiency using the asynchronous method, copyCGImageAtTime:actualTime:error:, which employs decoding efficiencies similar to those used during playback.

We will use generateCGImagesAsynchronouslyForTimes:completionHandler:to generate the images from video. It expects an NSArray of NSValue objects, each containing a CMTime, specifying the asset times at which we need an image.

Here’s a simple function that can be used to generate images asynchronously from the video:

-(void)generateImage
{
    NSURL *url = [NSURL fileURLWithPath:_videoPath];
    AVURLAsset *asset=[[AVURLAsset alloc] initWithURL:url options:nil];
    AVAssetImageGenerator *generator = [[AVAssetImageGenerator alloc] initWithAsset:asset];
    generator.appliesPreferredTrackTransform=TRUE;

    CMTime thumbTime = CMTimeMakeWithSeconds(30,30);
    
    AVAssetImageGeneratorCompletionHandler handler = ^(CMTime requestedTime, CGImageRef im, CMTime actualTime, AVAssetImageGeneratorResult result, NSError *error){
        if (result != AVAssetImageGeneratorSucceeded) {
            NSLog(@"couldn't generate thumbnail, error:%@", error);
        }
        // TODO Do something with the image
    };
    
    CGSize maxSize = CGSizeMake(128, 128);
    generator.maximumSize = maxSize;
    [generator generateCGImagesAsynchronouslyForTimes:[NSArray arrayWithObject:[NSValue valueWithCMTime:thumbTime]] completionHandler:handler];
    
}

Facebook Android SDK reauthorizeForPublish() loses old permissions

Mon, 14 Jan 2013 20:32:00 GMT

In Requesting extended permissions, I explained how to request extended permissions with the Facebook Android SDK. If you use this method to request different kinds of permissions at different stages, you would notice that making a call to [cci]session.reauthorizeForPublish()[/cci] with new permissions clears the already existing permissions from the session object and thus the call [cci]session.getPermissions()[/cci] would later return just the new permissions that are recently granted. This means that the Session object’s notion of its permissions is out of sync with the Facebook service and the access token associated with the Session still has those permissions and can still be used to make Graph API calls requiring any of the permissions it has been granted earlier (unless the user has subsequently revoked any of them, of course). This is a bug with the Facebook SDK.

An approach to work around this problem is to store the permissions that have been granted to us by persisting it in the Android Application class. The application class is the base class to maintain global application state. If you don’t already have an application class for your application, you can create one by extending Application.

public class MyApplication extends Application {

	Set<String> permissions;

	public Set<String> getPermissions(Collection<String> sessionPermissions) {
		if (permissions == null)
			permissions = new HashSet<String>();

		permissions.addAll(sessionPermissions);
		return permissions;
	}

	public void setPermissions(Set<String> permissions) {
		this.permissions = permissions;
	}

	public void addPermissions(Collection<String> newPermissions) {
		if (this.permissions == null) {
			this.permissions = new HashSet<String>();
		}
		this.permissions.addAll(newPermissions);
	}

	public void clearPermissions() {
		if (this.permissions != null)
			this.permissions.clear();
	}

}

<application
 android:icon="@drawable/icon"
 android:label="@string/app_name"
 android:name="MyApplication">

The first thing we need to do now is to obtain the current permissions granted by the user for our session when the app is launched (and user is already loggeed in) or when the user logs in using Facebook. I use the following method to retrieve current permissions from Facebook and persist them in the application object. This should be called in onSessionStateChanged when a new session is opened or in onCreate if the user is already logged in.

protected void retreiveCurrentPermissions(Session session) {
	FacebookClient fbClient = new DefaultFacebookClient(session.getAccessToken());
	JsonObject resultJson = fbClient.fetchObject("me/permissions", JsonObject.class);
	List<String> permissions = new ArrayList<String>();
	try {
		if (resultJson.getJsonArray("data").length() > 0) {
			JsonObject permissionsJson = resultJson.getJsonArray("data").getJsonObject(0);
			for (String key : JsonObject.getNames(permissionsJson)) {
				if (permissionsJson.getInt(key) == 1) {
					permissions.add(key);
				}
			}
		}
		((MyApplication) getApplication()).addPermissions(permissions);
	} catch (JsonException e) {
		e.printStackTrace();
	}
}

Another important point to keep in mind is to clear the permissions if the user logs out of Facebook. This can be done by calling ((MyApplication) getApplication()).clearPermissions(); in onSessionStateChanged when the session is closed.

Now we are persisting the session permissions in our application object. Before performing an operation that requires extended permission, we should now check if the required permissions are already present in ((MyApplication)getApplication()).getPermissions(session.getPermissions()); and make a Reauthorization request if this is not the case. In this case, we will then have to add the permissions in onSessionStateChanged if the state of the session is SessionState.OPENED_TOKEN_UPDATED.

Requesting extended permissions | Facebook SDK | Android

Wed, 09 Jan 2013 18:13:24 GMT

Facebook documentation focuses on the point that applications accessing user’s data from Facebook should not ask for any permission they are not going to make use of (especially publish_stream and offline_access). Here’s a quote from a Facebook document:

Depending on your application’s needs, you may need additional permissions from the user. A large number of calls do not require any additional permissions, so you should first make sure you need a permission. This is a good idea because this step potentially adds friction to the user’s process. Another point to remember is that this call can be made even after the user has first connected. So you may want to delay asking for permissions until as late as possible.

However, while working on an Android application that uses the Facebook SDK, I couldn’t find good documentation on how to do this. Here’s what I have been able to come up with by reading the code in sample Facebook applications in the SDK.

Let us first define what permissions we are going to need.

// List of additional write permissions being requested
private static final List<String> PERMISSIONS = Arrays.asList("publish_stream, publish_actions");

// Request code for reauthorization requests. 
private static final int REAUTH_ACTIVITY_CODE = 100; 

// Flag to represent if we are waiting for extended permissions
private boolean pendingAnnounce = false;

As you might have guessed from the permissions, we would be publishing some thing to user’s wall. Lets call this method that handles the sesison as publishAnnounce. We add our activity as the session callback for Facebook session updates. You can pass objects of any class that implements StatusCallback. We will see more about this later. Here’s what publishAnnounce would look like.

private void handleAnnounce() {
	pendingAnnounce = false;
	Session session = Session.getActiveSession();

	if (session == null || !session.isOpened()) {
		return;
	}

	List<String> permissions = session.getPermissions();
	if (!permissions.containsAll(PERMISSIONS)) {
		pendingAnnounce = true; // Mark that we are currently waiting for confirmation of publish permissions 
		session.addCallback(this); 
		requestPublishPermissions(this, session, PERMISSIONS, REAUTH_ACTIVITY_CODE);
		return;
	}

	// TODO: Publish the post. You would need to implement this method to actually post something
	publishMessage();
}

void requestPublishPermissions(Activity activity, Session session, List<String> permissions,
	int requestCode) {
	if (session != null) {
		Session.ReauthorizeRequest reauthRequest = new Session.ReauthorizeRequest(activity, permissions)
			.setRequestCode(requestCode);
		session.reauthorizeForPublish(reauthRequest);
	}
}

We first checked if we had all the required permissions. If not, we create a new ReauthorizeRequest with the set of permissions that we are requesting. This guides user outside our app to either the Facebook app or browser to confirm the permissions. When the user confirms the requests, he is redirected back to our application and onActivityResult for the Activity that invoked the reauthorization request is called. Let us see how it would look like:

@Override
public void onActivityResult(int requestCode, int resultCode, Intent data) {
	super.onActivityResult(requestCode, resultCode, data);
	switch (requestCode) {
	case REAUTH_ACTIVITY_CODE:
		Session session = Session.getActiveSession();
		if (session != null) {
			session.onActivityResult(this, requestCode, resultCode, data);
		}
		break;
	}
}

All we need to do here is to pass the result to the active Facebook session. This processes the result and calls the onSessionStateChanged if the session state changes. We then check in that method if our token was successfully updated and try to publish the message again.

protected void onSessionStateChange(final Session session, SessionState state, Exception exception) {
	if (session != null && session.isOpened()) {
		if (state.equals(SessionState.OPENED_TOKEN_UPDATED)) {
			// Session updated with new permissions
			// so try publishing once more.
			tokenUpdated();
		}
	}
}

/**
 * Called when additional permission request is copmleted successfuly.
 */
private void tokenUpdated() {
	// Check if a publish action is in progress
	// awaiting a successful reauthorization
	if (pendingAnnounce) {
		// Publish the action
		handleAnnounce();
	}
}

EDIT: For the new SDK(3.0.1), I have the following:

public static void requestPublishPermissions(Activity activity, Session session, List<String> permissions,
	int requestCode) {
	if (session != null) {
		Session.NewPermissionsRequest reauthRequest = new Session.NewPermissionsRequest(activity, permissions)
		.setRequestCode(requestCode);
		session.requestNewPublishPermissions(reauthRequest);
	}
}

public static void requestReadPermissions(Activity activity, Session session, List<String> permissions,
	int requestCode) {
	if (session != null) {
		Session.NewPermissionsRequest reauthRequest = new Session.NewPermissionsRequest(activity, permissions)
		.setRequestCode(requestCode);
		session.requestNewReadPermissions(reauthRequest);
	}
}

Extending RestFB model with extra fields

Thu, 03 Jan 2013 17:42:30 GMT

RestFB is a simple and flexible Facebook Graph API client for Java. I have used it recently for one of my projects and it greatly simplifies the graph requests. However, as the Graph API is constantly changing, it is really hard for the developers to keep updated with the latest additions in the graph api. I found that the Post model in RestFB representing the news feed posts didn’t have any story field to represent the story being posted on Facebook. RestFB makes it really simple to extend the existing models with extra information without having to change anything in the library code. To add the story field, I created a custom class PostWithStory containing the field. All we need to have the field is to create a new data member and annotate it with @Facebook.

import com.restfb.Facebook;
import com.restfb.types.Post;

public class PostWithStory extends Post {

    @Facebook("story")
    private String story;

    public String getStory() {
        return story;
    }

    public void setStory(String story) {
        this.story = story;
    }
}

Now, when retrieving the posts from news feed, I use the following to get my posts with the story field.

Connection<PostWithStory> myFeed = facebookClient.fetchConnection("me/home", PostWithStory.class);

If you want to have more control on the data that is being returned, you can have a method in the custom class annotated with @JsonMappingCopmleted and it will be called after RestFB finishes mapping the Jsonobject.

// You can annotate methods with @JsonMappingCompleted to perform
// post-mapping operations.
//
// This is useful if you want to massage the data FB returns.

@JsonMappingCompleted
void allDone(JsonMapper jsonMapper) {
  if(lotsOfNumbers.size() == 0)
    throw new IllegalStateException("I was expecting more numbers!");
}

Rendering photos and videos from Facebook news feed in Android application

Thu, 03 Jan 2013 09:24:16 GMT

In the last post, I described how to get photos and videos from a user’s Facebook news feed. In this post, I will continue on and describe how to build a simple user interface for showing these photos and videos for an Android app. Let’s have a look at how this is going to look like:

Let us first build a model that would describe the main parts of data that we obtained from the fql queries. We will extract the main parts in the media objects as follows:

if (mediaResultObject.has("attachment") && mediaResultObject.getJsonObject("attachment").has("media")) {
	// Get attachment name and href
	String attachmentName = Utils.getStringFromJsonIfExists(mediaResultObject.getJsonObject("attachment"),
			"name");
	final String attachmentHref = Utils.getStringFromJsonIfExists(
			mediaResultObject.getJsonObject("attachment"), "href");

	// Get id of actor who posted the content
	String actorId = mediaResultObject.getString("actor_id");
	String postId = mediaResultObject.getString("post_id");
	String createdTime = mediaResultObject.getString("created_time");

	JsonArray mediaArray = mediaResultObject.getJsonObject("attachment").getJsonArray("media");
	for (int i = 0; i < mediaArray.length(); ++i) {
		final JsonObject mediaObject = mediaArray.getJsonObject(i);

		String type = mediaObject.getString("type");

		// Get thumbnail url
		String thumbnailUrl = Utils.getStringFromJsonIfExists(mediaObject, "src");

		MediaPost post = new MediaPost(attachmentName, attachmentHref, actorId, postId, createdTime, type,
				thumbnailUrl);

		String fbid = null;
		if (type.equals("photo") && mediaObject.has("photo")) { // Photo
			fbid = mediaObject.getJsonObject("photo").getString("fbid");
		} else if (mediaResultObject.getJsonObject("attachment").has("fb_object_id")) {
			fbid = mediaResultObject.getJsonObject("attachment").getString("fb_object_id");
		}
		post.setHref(mediaObject.getString("href"));
		post.setFbId(fbid);
		post.setMediaObject(mediaObject);
		post.setAlt(mediaObject.getString("alt"));

		posts.add(post);
	}
}

The mediaObject here is a jsonObject parsed from a json object like so:

{
  "actor_id": 202209223157016,
  "created_time": 1357059211,
  "likes": {
    "href": "https://www.facebook.com/browse/likes/?id=501300199914582",
    "count": 13,
    "sample": [
      100001172558908,
      202209223157016,
      100002876441601
    ],
    "friends": [
      100000185344749
    ],
    "user_likes": false,
    "can_like": true
  },
  "post_id": "202209223157016_501300213247914",
  "attachment": {
    "media": [
      {
        "href": "https://www.facebook.com/photo.php?fbid=501300199914582&set=a.202685659776039.53449.202209223157016&type=1&relevant_count=1",
        "alt": "I am alone but not dead ... rnrnrn~ Alone Tree Silhouette at Railay Beach",
        "type": "photo",
        "src": "https://fbcdn-photos-a.akamaihd.net/hphotos-ak-prn1/12594_501300199914582_1535726991_s.jpg",
        "photo": {
          "aid": "202209223157016_53449",
          "pid": "202209223157016_1613061",
          "fbid": 501300199914582,
          "owner": 202209223157016,
          "index": 1,
          "width": 634,
          "height": 1024
        }
      }
    ],
    "name": "",
    "caption": "",
    "description": "",
    "properties": [
    ],
    "icon": "https://fbstatic-a.akamaihd.net/rsrc.php/v2/yz/r/StEh3RhPvjk.gif",
    "fb_object_type": "photo",
    "fb_object_id": "202209223157016_1613061"
  }
}

Here’s the layout we will use for each object:

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="https://schemas.android.com/apk/res/android"
    android:layout_width="fill_parent"
    android:layout_height="wrap_content"
    android:background="@drawable/gradient_background_gray"
    android:paddingLeft="5dp"
    android:paddingRight="5dp" >

    <ImageView
        android:id="@+id/imageView"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_alignParentLeft="true"
        android:layout_centerVertical="true"
        android:src="@drawable/abs__ab_bottom_solid_dark_holo" >
    </ImageView>

    <LinearLayout
        android:id="@+id/linearLayout1"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_alignParentLeft="true"
        android:layout_alignParentTop="true"
        android:background="@color/TranslucentBlack"
        android:orientation="vertical" >

        <TextView
            android:id="@+id/attachment_name"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_gravity="center_horizontal"
            android:gravity="center_horizontal"
            android:text="Medium Text"
            android:textAppearance="?android:attr/textAppearanceMedium"
            android:textColor="@color/facebookName"
            android:textStyle="bold" />

    </LinearLayout>

    <LinearLayout
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_alignParentBottom="true"
        android:layout_alignParentLeft="true"
        android:background="@color/TranslucentBlack"
        android:orientation="vertical" >

        <TextView
            android:id="@+id/picture_title"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_gravity="center_horizontal"
            android:ellipsize="end"
            android:gravity="center_horizontal"
            android:maxLines="3"
            android:text="Medium Text"
            android:textAppearance="?android:attr/textAppearanceMedium"
            android:textColor="@color/WhiteSmoke" />

        <TextView
            android:id="@+id/user_name"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:text="Medium Text"
            android:textColor="@color/facebookName"
            android:layout_gravity="right"
            android:textAppearance="?android:attr/textAppearanceSmall" />
    </LinearLayout>

    <ImageButton
        android:id="@+id/playButton"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_centerHorizontal="true"
        android:layout_centerVertical="true"
        android:src="@drawable/ic_play" />

</RelativeLayout>

We will use a custom PagerAadpter to render each item in the page. Here’s the function that would then instantiate each view in the PagerAdapter.

public Object instantiateItem(View collection, int position) {
	LayoutInflater inflater = (LayoutInflater) collection.getContext().getSystemService(
			Context.LAYOUT_INFLATER_SERVICE);
	View view = inflater.inflate(R.layout.view_image_photo, null);
	final MediaPost post = posts.get(position);

	String thumbnailUrl = post.getThumbnailUrl();

	if (post.getType().equals("video")) {
		// Overlay play button and handle click.
		ImageButton imageButton = (ImageButton) view.findViewById(R.id.playButton);
		imageButton.setOnClickListener(new OnClickListener() {
			@Override
			public void onClick(View v) {
				playVideo(post.getMediaObject().getJsonObject("video"));
			}
		});
	} else if (post.getType().equals("photo")) {
		// Hide play button
		view.findViewById(R.id.playButton).setVisibility(View.GONE);

		// Get better thumbnail url.
		thumbnailUrl = "https://graph.facebook.com/" + post.getFbId() + "/picture?type=normal&access_token="
				+ accessToken;
	}

	// Load thumbnail image in image view
	if (thumbnailUrl != null) {
		ImageView imageView = (ImageView) view.findViewById(R.id.imageView);
		imageLoader.DisplayImage(thumbnailUrl, imageView, false);
	}

	// Add details above and below
	TextView mediaName = (TextView) view.findViewById(R.id.picture_title);
	TextView attachmentNameTextView = (TextView) view.findViewById(R.id.attachment_name);
	TextView userNameTextView = (TextView) view.findViewById(R.id.user_name);

	// Set user name or hide view.
	if (post.getActorId() != null) {
		userNameTextView.setText(users.get(post.getActorId()).getString("name"));
	} else {
		userNameTextView.setVisibility(View.GONE);
	}

	mediaName.setText(post.getAlt());
	if (post.getAttachmentName() != null && post.getAttachmentName().length() > 0) {
		attachmentNameTextView.setText(post.getAttachmentName());
		if (post.getAttachmentHref() != null) {
			((View) attachmentNameTextView.getParent()).setOnClickListener(new OnClickListener() {
				@Override
				public void onClick(View v) {
					Intent browserIntent = new Intent(Intent.ACTION_VIEW, Uri.parse(post.getAttachmentHref()));
					context.startActivity(browserIntent);
				}
			});
		}
	} else {
		attachmentNameTextView.setVisibility(View.GONE);
	}

	((ViewPager) collection).addView(view, 0);
	return view;
}

Playing the video from the url on Android can be a bit tricky. One thing that I found about streaming videos on Android is that it is difficult to stream videos with url beginning with https. Fortunately, for the videos that we receive from facebook, just converting the urls to http works fine. Here’s how you would play the video in the default media player in android.

protected void playVideo(JsonObject video) {
	String videoUrl = Utils.getStringFromJsonIfExists(video, "source_url");
	try {
		if (videoUrl.startsWith("https")) {
			videoUrl = "http" + videoUrl.substring(5);
		}
		// First try to obtain correct mime type and play video.
		Intent intent = new Intent(Intent.ACTION_VIEW);
		String extension = android.webkit.MimeTypeMap.getFileExtensionFromUrl(videoUrl);
		String mimetype = android.webkit.MimeTypeMap.getSingleton().getMimeTypeFromExtension(extension);
		intent.setDataAndType(Uri.parse(videoUrl), mimetype);
		context.startActivity(intent);
	} catch (ActivityNotFoundException e) {
		// Unable to play using mime type.
		Intent intent = new Intent(Intent.ACTION_VIEW);
		intent.setData(Uri.parse(Utils.getStringFromJsonIfExists(video, "display_url")));
		context.startActivity(intent);
	}

}

We are almost done. Now we just need to create a ViewPager and set the adapter. Create the view pager in the layout and then get the reference to the ViewPager in your Activity/Fragment and set the subclass of PagerAdapter you just created.

<android.support.v4.view.ViewPager
  android:id="@+id/viewPager"
  android:layout_width="match_parent"
  android:layout_height="match_parent" />

Getting all photos and videos from Facebook news feed

Wed, 26 Dec 2012 11:34:11 GMT

I have been working with the Facebook API recently and I came across the need to get all the photos and videos in the news feed. Given the comprehensive Graph API, I had initially thought that it would be straight forward getting these details from the news feed. After struggling a bit with the available API methods, I decided that FQL might be the best way to go about this. And it turned out that doing this with FQL wasn’t all that difficult. Here’s a simple FQL query for getting all the photos and videos in the news feed. stream_filter contains the filters for filtering the stream of posts. This contains several filters that include the lists that the user has built and some default filters like the ones for photos and videos.

SELECT actor_id, created_time, likes, post_id, attachment
FROM stream
WHERE filter_key IN (
  SELECT filter_key FROM stream_filter WHERE uid = me() AND (name = "Photos" OR name = "Video")
)

You can have a look at the results of the query using the Graph API Explorer. This gives us the actor_id which is the id of the user/page on Facebook that has shared the post, the likes on the post, the id for the post and an attachment that contains more details about the media objects included in the posts. In order to get the details for the users who have shared the posts, we need to use multiple fql queries to query the profile table for user details.

{
	media: 'SELECT actor_id, created_time, likes, post_id, attachment FROM stream WHERE filter_key IN ( SELECT filter_key FROM stream_filter WHERE uid = me() AND (name = "Photos" OR name = "Video"))',
	users: 'SELECT id, name, url, pic FROM profile WHERE id IN (SELECT actor_id FROM #media)'
}

I will describe in the next post how we can parse the results from these queries to build a simple screen in Android with the results that allow users to browse through these photos and videos.

Building an extensible news feed reader

Fri, 23 Nov 2012 17:46:55 GMT

We built a news reader client for Android using Titanium called hackurls which shows the top news from some of the best technology news sharing websites. How long it took us? Less than a day. Although, I wouldn’t call the UI for the client to be very good, I find the app in itself to be very useful. Building this was fun and it wasn’t overly complicated. We built simple parsers for the feeds to get them to a common format so that our rendering could be done easily. This also leaves us a lot of space to improve the user experience while keeping most of our code for the backend un touched. Here’s an example for the service that retrieves news from Hackernews.

function HackerNews() {

    var STR_HN_URL = 'https://hndroidapi.appspot.com/news/format/json/page/?appid=YOUR_APP_ID';
    var STR_HN_BASE = 'https://news.ycombinator.com/';

    /**
     * Gets the domain of a url
     */
    function getDomain(url) {
        var match = url.match(/:\/\/(.[^/]+)/);
        if(match === null) {
            return ' ';
        }
        return (match[1]).replace('www.', '');
    }


    /**
     * Gets the post from Hackernews.
     */
    this.getPosts = function(callback) {
        var xhr = Titanium.Network.createHTTPClient();

        xhr.onload = function() {
            Ti.API.info(this.responseText);
            var parsedJson = JSON.parse(this.responseText);

            if(parsedJson !== undefined && parsedJson.items !== undefined) {
                var items = [];
                for(var i = 0, j = parsedJson.items.length; i < j; i++) {
                    var post = parsedJson.items[i];
                    var domain = getDomain(post.url);
                    items.push({
                        title : post.title,
                        url : (domain == ' ') ? STR_HN_BASE + post.url : post.url,
                        id : post.item_id,
                        commentCount : post.comments,
                        time : post.time,
                        timetype : 'pretty',
                        details : domain,
                        user : post.user,
                        points : post.score,
                        description : post.description
                    });
                };
                callback.call(this, items);
            }
        };
        // open the client
        xhr.open('GET', STR_HN_URL);

        // send the data
        xhr.send();

    };
};

module.exports = HackerNews;

Here’s another module that parses data for posts from Reddit and Proggit.

var STR_REDDIT_URL = 'https://www.reddit.com/hot.json';
var STR_PROGGIT_URL = 'https://www.reddit.com/r/programming/hot.json';

function Reddit(url, type) {
	var STR_REDDITBASE = 'https://www.reddit.com'
	this.getPosts = function(callback) {
		var xhr = Titanium.Network.createHTTPClient();

		xhr.onload = function() {
			var parsedJson = JSON.parse(this.responseText);
			if(parsedJson && parsedJson.data && parsedJson.data.children) {
				var items = [];
				for(var i = 0, j = parsedJson.data.children.length; i < j; i++) {
					var postData = parsedJson.data.children[i].data;
					items.push({
						title : postData.title,
						url : STR_REDDITBASE + postData.permalink,
						id : postData.id,
						commentCount : postData.num_comments,
						points : postData.score,
						time : postData.created_utc,
						timetype : 'timestamp',
						user : postData.author,
						thumbnail : postData.thumbnail,
						details : (type==='proggit')?postData.domain:postData.subreddit
					});
				};
				Ti.API.log(items);
				callback.call(this, items);
			}
		};
		// open the client
		xhr.open('GET', url);

		// send the data
		xhr.send();

	};
};

module.exports = Reddit;

Once we have all the posts with the details that we have here, building an amazing UI is the only thing that remains which I will leave up to you. There are similar parsers that we made for Slashdot, Techmeme, Wired and Dzone. If you want to give this a try, feel free to fork Hackurls on github or get Hackurls on Play store.

Managing multiple languages with Appcelerator Titanium

Sat, 17 Nov 2012 14:44:26 GMT

Titanium provides a number of JavaScript functions in the Titanium.Locale namespace for use in localization. It also provides String formatting functions to handle dates, times, currencies, and more. You can even internationalize the name of your app itself. There’s a good tutorial on how to do this here. Titanium also allows you to define internationalization files and automatically chooses the correct one based on device locale settings. But what if you want to allow the user to choose the language for the app from within the application settings? To provide this functionality in an app built with Titanium, we would need to override the function L() that Titanium uses to retrieve localised strings from one of the xml files. Here, we will parse the xml files manually based on the user’s selected language and find the localised string that user is looking for.

/**
 * Returns the text based on the selected language in the application settings.
 */
function L(text) {
  if (Ti.App.languageXML === undefined || Ti.App.languageXML === null) {
    var langFile = Ti.App.Properties.getString('SETTING_LANGUAGE'); // We should store user's language setting in SETTING_LANGUAGE
    var file = Ti.Filesystem.getFile(Ti.Filesystem.resourcesDirectory, 'i18n/' + langFile + '.xml'); // Get the corresponding file from i18n
    if (!file.exists()) {
      var langFile = "en"; // Fall back to english as the default language
      file = Ti.Filesystem.getFile(Ti.Filesystem.resourcesDirectory, 'i18n/' + langFile + '.xml');
    }
    var xmltext = file.read().text;
    var xmldata = Titanium.XML.parseString(xmltext); // Parse the xml
    Ti.App.languageXML = xmldata; // Store the parsed xml so that we don't parse everytime L() is called
  }
  // Get the localised string from xml file
  var xpath = "/resources/string[@name='" + text + "']/text()"; 
  var result = Ti.App.languageXML.evaluate(xpath).item(0);
  if (result) {
    return result.text;
  } else {
    return text; // Return the text if localised version not found
  }
}

Now, all you need is to include this in every file and whenever you want to use a localised string, use L('my-string-here') in its place. Don’t forget to set the currently selected language using Ti.App.Properties.setString('SETTING_LANGUAGE','en'); All that’s left now is to create your internationalisation files in i18n directory inside your app resources and declare strings in the following format:

<?xml version="1.0" encoding="UTF-8"?>
<resources>
  <string name="home">Home</string>
  <string name="settings">Settings</string>
</resources>

Moving content located under the iOS keyboard

Mon, 12 Nov 2012 17:23:48 GMT

The iOS keyboard slides from the bottom of the screen and is positioned on the top of application's content. Apple wrote a guide for handling such events and moving the content from behind the keyboard. The basic workflow for this is to listen to the keyboard notifications, get the size of the keyboard, adjust the bottom content inset of scroll view by the keyboard height and scroll the target text field into view.

// Call this method somewhere in your view controller setup code.
- (void)registerForKeyboardNotifications
{
    [[NSNotificationCenter defaultCenter] addObserver:self
            selector:@selector(keyboardWasShown:)
            name:UIKeyboardDidShowNotification object:nil];

   [[NSNotificationCenter defaultCenter] addObserver:self
             selector:@selector(keyboardWillBeHidden:)
             name:UIKeyboardWillHideNotification object:nil];

}

// Called when the UIKeyboardDidShowNotification is sent.
- (void)keyboardWasShown:(NSNotification*)aNotification
{
    NSDictionary* info = [aNotification userInfo];
    CGSize kbSize = [[info objectForKey:UIKeyboardFrameBeginUserInfoKey] CGRectValue].size;

    UIEdgeInsets contentInsets = UIEdgeInsetsMake(0.0, 0.0, kbSize.height, 0.0);
    scrollView.contentInset = contentInsets;
    scrollView.scrollIndicatorInsets = contentInsets;

    // If active text field is hidden by keyboard, scroll it so it's visible
    // Your application might not need or want this behavior.
    CGRect aRect = self.view.frame;
    aRect.size.height -= kbSize.height;
    if (!CGRectContainsPoint(aRect, activeField.frame.origin) ) {
        CGPoint scrollPoint = CGPointMake(0.0, activeField.frame.origin.y-kbSize.height);
        [scrollView setContentOffset:scrollPoint animated:YES];
    }
}

// Called when the UIKeyboardWillHideNotification is sent
- (void)keyboardWillBeHidden:(NSNotification*)aNotification
{
    UIEdgeInsets contentInsets = UIEdgeInsetsZero;
    scrollView.contentInset = contentInsets;
    scrollView.scrollIndicatorInsets = contentInsets;
}

There is one important point that Apple misses here in the guide though. This doesn't work very well when the app is operating in the landscape mode. I struggled with this for quite some time before actually figuring out that the keyboard height is always returned for the portrait mode irrespective of the orientation that the app is currently operating in. Here's a quick hack to fix this.

if (UIInterfaceOrientationIsLandscape([UIDevice currentDevice].orientation)) {
    kbSize = CGSizeMake(kbSize.height, kbSize.width);
}

Move, Copy or swap DOM elements?

Mon, 03 Sep 2012 14:24:50 GMT

If you have been reading my posts, you probably already know that I worked on jMatrixBrowse this summer. For those who don't know, it is a jQuery plugin that allows creating browsable matrices (just like Google maps) on the web. See a quick demo here.

An important objective for the plugin was not to make a lot of requests to the backend to obtain the data in the matrix. However, we still wanted to have the data available for the user as soon as he reaches a particular cell. We implemented a background caching module for this that loaded all the cells in the matrix in a separate background container so that all the data is available in the DOM when it is required. This was implemented as a completely separate module which handled all calls to the backend. So the main plugin only made requests to the background loading module and it decided which cells it already had and which it needed to retrieve from the API on the go and sent the results to the renderer.

This was working great. The only thing that we needed to take care of was how to swap in the content from the background to the matrix content. Here are some experimental results from three strategies that we tested.

Strategy 1: Load the background data in a separate container and when more cells are required in the matrix, we clone and move the cells from the background container to the main matrix container at the corresponding position.
Strategy 2: Load the background data in the same container as the matrix content but make it initially hidden. Change the top/left of the background cells and show them when new data is required in the matrix.
Strategy 3: Load the background data in a separate container and when new data is required in the matrix, swap the html of the new data in place of the already existing cells.

|  Reload window   | Strategy  | Average Time (in msec)|
| One Page (5x10)  |     1     |            17         |
| One Page (5x10)  |     2     |             8         |
| One Page (5x10)  |     3     |             4         |
| One Row  (1x10)  |     1     |             4         |
| One Row  (1x10)  |     2     |             1.5       |
| One Row  (1x10)  |     3     |             1         |

As expected the first strategy is the slowest because cloning DOM elements is slow. Strategy 2 was the next slowest because moving content in the DOM is slow as well. The fastest was swapping the HTML content which required just about 1ms for loading one row of cells which is a good response time.

If you want to have a detailed look at how these strategies were implemented, have a look at the jMatrixBrowse code on github.

Handle keyboard shortcuts with jQuery

Tue, 28 Aug 2012 20:55:30 GMT

When you start building an application for the web, one of the features that almost always gets overlooked is accessibility. There are lots of plugins out there and most of them just ignore this. During the summer of 2011, I worked as a Google Summer of Code student for Inclusive Design Institute and the most unique thing about them was their focus on accessibility. I developed an image editor [source][<a href="https://pulkitgoyal.in/imageeditor/">demo</a>] component for the web. One of my focus during the development was to allow editing the images on the web without the use of mouse.

During this summer, I worked with NESCent on the development of jMatrixBrowse[source][<a href="https://pulkitgoyal.in/demo/jmatrixbrowse/" target="_blank">demo</a>]. This summer as well, I focused quite a lot on the keyboard accessibility for jMatrixBrowse. In this post, I’ll walk you through the process of creating keyboard shortcuts for jQuery plugins. And its really simple, thanks to jquery.hotkeys maintained by jeresig, the creator of jQuery.

The first step is to download the plugin. Its just one file. Get this. Now, its really simple to add keyboard shortcuts. Just include this script in the html. Here’s how you would now bind these in your js.

jQuery(document).bind('keydown', 'right', function(e) {
  // right arrow key pressed
});
jQuery(document).bind('keydown', 'left', function(e) {
  // left arrow key pressed
});
jQuery(document).bind('keydown', 'up', function(e) {
  // up arrow key pressed
});
jQuery(document).bind('keydown', 'down', function(e) {
  // down arrow key pressed
});
jQuery(document).bind('keydown', 'ctrl+up', function() {
  // ctrl + up arrow key pressed
});

Just add any combination of shortcuts that you want to bind to. Keep in mind not to use common browser shortcuts because the browsers might handle these shortcuts on their own and not pass them to your page. Firefox seems to be the most liberal in this regard and Safari the most restricting. So I would test the shortcuts in Safari to see if these are actually passed to my script.

Custom jQuery UI Draggable bounds

Sat, 25 Aug 2012 16:36:05 GMT

jQuery-UI draggable is very good for creating draggable components. It was perfect for the creating jMatrixBrowse. jMatrixBrowse is a jQuery plugin that allows creating large browsable matrices (like Google Maps) in the browser. jQuery UI Draggable, like the other jQuery UI widgets, is highly customisable. But I feel that the customisability is still a bit under documented. You can find a whole lot of new customisation possibilities if you dive into its code. For the project, I required a custom containment handler to be called to decide if the containment relation is satisfied or not. jQuery UI Draggable already gives a containment option but that only allows to contain the draggable element inside an already existing element or by giving absolute bounds. I wanted to check these bounds based on the current matrix state.

Draggable uses the generatePosition method to generate the new position for the draggable. To check custom bounds, I extended the generatePosition method and implemented by custom bounds checks in this.

```javascript // Override the original _generatePosition in draggable to check for matrix bounds on drag. dragContainer.draggable().data("draggable")._generatePosition = function(event) { return generatePositionsForDrag(dragContainer.draggable().data("draggable"), event); }; ```

First, we need to override the _generatePosition for our draggable element. Next, we will implement the new generate position similar to the one in draggable just using our bounds instead.

/**
* Generate new positions of the draggable element. This is used to override
* the original generate positions which had no way of specifying dynamic
* containment. This checks if the drag is valid by looking at the matrix
* coordinates and returns the new top and left positions accordingly.
*
* @param {Object} draggable - draggable object data.
* @param {Object} event - event that initiated the drag.
* @returns {Object} positions - new position of the draggable.
*/
function generatePositionsForDrag(draggable, event) {
  var o = draggable.options, scroll = draggable.cssPosition == 'absolute' && !(draggable.scrollParent[0] != document && $.ui.contains(draggable.scrollParent[0], draggable.offsetParent[0])) ? draggable.offsetParent : draggable.scrollParent, scrollIsRootNode = (/(html|body)/i).test(scroll[0].tagName);
  var pageX = event.pageX;
  var pageY = event.pageY;

  var newPosition = {
    top: (
            pageY															// The absolute mouse position
            - draggable.offset.click.top												// Click offset (relative to the element)
            - draggable.offset.relative.top												// Only for relative positioned nodes: Relative offset from element to offset parent
            - draggable.offset.parent.top												// The offsetParent's offset without borders (offset + border)
            + (jQuery.browser.safari && jQuery.browser.version < 526 && draggable.cssPosition == 'fixed' ? 0 : ( draggable.cssPosition == 'fixed' ? -draggable.scrollParent.scrollTop() : ( scrollIsRootNode ? 0 : scroll.scrollTop() ) ))
    ),
    left: (
            pageX															// The absolute mouse position
            - draggable.offset.click.left												// Click offset (relative to the element)
            - draggable.offset.relative.left												// Only for relative positioned nodes: Relative offset from element to offset parent
            - draggable.offset.parent.left												// The offsetParent's offset without borders (offset + border)
            + (jQuery.browser.safari && jQuery.browser.version < 526 && draggable.cssPosition == 'fixed' ? 0 : ( draggable.cssPosition == 'fixed' ? -draggable.scrollParent.scrollLeft() : scrollIsRootNode ? 0 : scroll.scrollLeft() ))
    )
  };

  // Impose contraints on newPosition to prevent crossing of matrix bounds. 
  // Compute change in position for the drag.
  var changeInPosition = {
    top: (draggable._convertPositionTo("absolute", newPosition).top - draggable.positionAbs.top),
    left: (draggable._convertPositionTo("absolute", newPosition).left - draggable.positionAbs.left)
  };

  return checkPositionBounds(newPosition, changeInPosition, draggable);
}

/**
 * Checks the bounds for the matirx from four directions to find if the bounds are violated and returns the new positions.
 * @param  {Object} newPosition - The new position of the container for which to check the bounds.
 * @param  {Object} changeInPosition - The change in position. {top:0, left:0} can be passed here.
 * @param  {Object} draggable - The draggable instance.
 * @returns {Object} newPosition of the container.
 */
function checkPositionBounds (newPosition, changeInPosition, draggable) {
  
  var firstRow = (_self.currentCell.row == 0)?1:0;
  var firstCol = (_self.currentCell.col == 0)?1:0;
  // Get element and container offsets
  var element = jQuery(_cellElements[firstRow][firstCol]);
  var containerOffset = _container.offset();
  var elementOffset = element.offset();

  // If we are at the topmost cell, then check that bounds from the top are maintained.
  if (_self.currentCell.row <= 1) {
    // The new posoition.top of the first element relative to cotainter.
    var top = changeInPosition.top + elementOffset.top - containerOffset.top;
    if (top > 0) { // The drag crosses matrix bounds from the top.
      newPosition.top = newPosition.top - top;
    }
  }

  // If we are at the leftmost cell, then check that bounds from the left are maintained.
  if (_self.currentCell.col <= 1) {
    // The new posoition.top of the first element relative to cotainter.
    var left = changeInPosition.left + elementOffset.left - containerOffset.left;
    if (left > 0) { // The drag crosses matrix bounds from the left.
      newPosition.left = newPosition.left - left;
    }
  }

  // Get element offset for last element
  element = jQuery(_cellElements[_cellElements.length-1][_cellElements[0].length-1]);
  elementOffset = element.offset();

  // If we are at the bottomost cell, then check that bounds from the bottom are maintained.
  if (_self.currentCell.row - _configuration.getNumberOfBackgroundCells() + _cellElements.length - 1 >= _api.getMatrixSize().height-1) {
    var containerBottom = (containerOffset.top + _container.height());
    var elementBottom = (changeInPosition.top + elementOffset.top + element.height());
    // The new posoition.bottom of the last element relative to cotainter.
    var bottom =  containerBottom - elementBottom;
    if (bottom > 0) { // The drag crosses matrix bounds from the bottom.
      newPosition.top = newPosition.top + bottom;
    }
  }

  // If we are at the leftmost cell, then check that bounds from the left are maintained.
  if (_self.currentCell.col - _configuration.getNumberOfBackgroundCells() + _cellElements[0].length - 1 >= _api.getMatrixSize().width-1) {
    // The new posoition.right of the first element relative to cotainter.
    var containerRight = (containerOffset.left + _container.width());
    var newElementRight = (changeInPosition.left + elementOffset.left + element.width());
    var right =  containerRight - newElementRight;
    if (right > 0) { // The drag crosses matrix bounds from the left.
      newPosition.left = newPosition.left + right;
    }
  }

  return newPosition;
}

All you need to check your own bounds now for draggable is to implement your own checkPositionBounds and you are good to go. I almost decided to drop using the jQuery UI draggable and create my own. But in the end, a close look at the draggable code saved me the pain to write a custom draggable. So, now if you ever need to implement something complicated with jQuery UI widget, don't just try creating your own clone. The functionality might itself be buried into the code, you just need to figure out where to look.

Update UI for MacBook Pro Retina display

Sat, 18 Aug 2012 08:48:08 GMT

Recently I purchased a MacBook Pro with Retina display and believe me Apple has done an incredible job with the next-gen MacBook Pro. The Retina display is a powerhouse. Almost everything else in native apps and photos looks amazingly crisp and clear on the Retina display. But for me as a developer when I started Eclipse first time, retina UI was not supporting Eclipse display very well. So here is the workaround to update UI:

Go to Contents->Info.plist by clicking on “Show package contents” on the Eclipse.app. Then Just above

</dict>
</plist>

Place this:

<key>NSHighResolutionCapable</key>
<true/>

Then, log out and restart the Eclipse. This also works for Aptana as well as for Titanium Studio and in general for all IDEs that are built on top of Eclipse.

Writing scalable recommender system with Hadoop

Wed, 01 Aug 2012 22:39:33 GMT

Lets say we want to implement a very simple “people you might know” recommender algorithm. Obviously, the speed and scalability of such an algorithm is as important as the actual logic behind the algorithm because such algorithms generally run over a “huge” graph and implementing these normally would probably take a lot of time for recommending friends even for just one user. I will show how to implement a very naive recommender algorithm efficiently. The basic idea of the algorithm is that if person A and person B do not know each other but they have a lot of mutual friends, then the system should recommend that they connect with each other. We will do this is hadoop which is an open source software for highly reliable, scalable distributed computing. The main idea behind it is map and reduce jobs. I assume that you are already familiar with these concepts (tutorial). If not, I wouldn’t recommending reading this post further without having a look at what a mapper and reducer is.

In order to recommend such friends, we first need to count the number of mutual friends that each pair of persons have in such a network. For this, we will need a map reduce job that functions similar to the map-reduce job for finding the frequency of words in a file. For every pair of friends in a list, we output the following tuple: <[friend1 friend2 1], 1> . In addition to this, we also output a tuple <[source friend -1], -1> for every pair of persons who are friends. i.e. For an input:

source -> {“friend1”, “friend2”, “friend3”}

the mapper outputs

{[source, friend1, -1], -1}; {[source, friend2, -1], -1};
{[source, friend3, -1], -1}; {[friend1, friend2, 1], 1};
{[friend1, friend3, 1], 1}; {[friend2, friend3, 1], 1}

Now, we need to define our equals() method such that it can looks at the two names in the key to find equal keys. We also need to define our Comparator such that it sorts the keys based on the third attribute in the key. Therefore, the reducer gets as input a key denoting a pair of friends along with the list of their number of common friends. In the reduce function, we can check if the first record has a -1 in the key. If there is such a record, we can ignore that pair of friends because they already have a direct connection between them. Finally, we aggregate the values for all the keys in the reducer and output the tuple <[friend1, friend2], numberOfMutualFriends> for all the pairs of friends which don’t have third attribute as -1 in the key.

function map(input)
  split input to obtain source and friends[]
  for (i = 1:n)
    f1 = friends[i]
    output <[source, f1, -1], -1>
    for (j = i:n)
      f2 = friends[j]
      output <[f1, f2, 1], 1
    end
  end
end

function equals(key1, key2)
  if ((key1[1] == key2[1] && key1[2] == key2[2]) || key1[1] == key2[2] && key1[2] == key2[1])
    return true;
  end
  return false;
end

function compare(key1, key2)
  return (key1[3] < key2[3]);
end

function reduce(key, values)
  if (values[0] == -1)
    return;
  end
  numberOfCommonFriends = 0;
  for (value v in values)
    numberOfCommonFriends += v;
  end
  output <[key[1], key[2]], numberOfCommonFriends>
end

After the first map-reduce job, we obtain a list of pair of persons along with the number of common friends that they have. Our final map-reduce job looks at this list and outputs a list of persons they have maximum number of common friends with. Hence, our map job just outputs<[pair[1], numberOfCommonFriends], pair[2]> and<[pair[2], numberOfCommonFriends], pair[1]>. Our equals() method would look at the person in the key to find equal keys and our comparator would look at the numberOfCommonFriends to sort the keys. This would ensure that the tuples for the same person go to the same reducer and in a sorted order by the number of common friends. Our reducer then just needs to look at the top 10 values and output the list.

function map(input)
  split input to obtain friend1, friend2 and numberOfCommonFriends
  output <[friend1, numberOfCommonFriends], friend2>
  output <[friend2, numberOfCommonFriends], friend1>
end

function equals(key1, key2)
  if (key1[1] == key2[1])
    return true;
  end
  return false;
end

function compare(key1, key2)
  return (key1[2] < key2[2]);
end

function reduce(key, values)
  suggestions = []
  for (i = 1:min(10,size(values)))
    suggestions.push(values[i]);
  end
  output <key[1], suggestions>
end

This is all we need to do to have our small recommender system up and running on Hadoop which will already scale quite well. There are still plenty of things that can be done here both in the algorithm as well as the implementation to improve the results and scalability.

Link order for libraries C and C++

Tue, 31 Jul 2012 21:30:24 GMT

There are several advantages of using external libraries in C or C++. It saves a lot of effort as there are several wonderful C++ libraries out there to do just about anything. But there is one problem that many beginners face when beginning to use libraries. undefined reference to <function>.

Lets say we have a program that uses functions defined in an external library mylib. We can link it by adding -lmylib when we compile our code. So we do this:

g++ -lmylib main.c -o main    (incorrect order)

main.o: In function `main':
main.o(.text+0xf): undefined reference to `myfunc'

But mylib is linked and it has a definition for myfunc. The problem here is that the order of linking is important. The traditional behavior of linkers is to search for external functions from left to right in the libraries specified on the command line. This means that a function in libraryA is referenced from libraryB, libraryA should be linked after libraryB. This might often lead to circular dependencies when a library uses some function from another library which in turn uses a function from the first library. For example, if foo uses a function defined in bar and bar uses a function defined in foo, we would need to write:

g++ -o main -lfoo -lbar -lfoo

Although, this works well in practice and is used quite frequently, a much better solution to the above problem is to use the linker’s —start-group and —end-group options, like this:

g++ -o main -Wl,--start-group -lfoo -lbar -Wl,--end-group

Documenting jQuery plugins with jsdoc-toolkit

Mon, 04 Jun 2012 07:09:24 GMT

Documenting a code is very important, specially when you are working in a team. For java, JavaDoc has been my favorite. But there seems to be a lack of documentation tools for Javascript. The one that I found most easy to use (transitioning from JavaDoc) was jsdoc-toolkit. But since there was no good tutorial for how to begin documenting jQuery plugins with jsdoc, I decided to write one.

Installation in Ubuntu is simple. Just use apt-get install.

sudo apt-get install jsdoc-toolkit

I won’t go into the details for other platforms but it should be pretty simple to install on other platforms as well. To generate the documentation, use:

jsdoc -d=./doc ./src

This would generate the documentation in doc directory for the files in src. The main issues start from here because when you try to document your jQuery plugin using this, it doesn’t recognize the friendly $. Nor does it know what jQuery is.

Trying to document jMatrixBrowse as a member of undocumented symbol jQuery
Trying to document jMatrixBrowse as a member of undocumented symbol jQuery.fn

Trying to document jMatrixBrowse as a member of undocumented symbol $
Trying to document jMatrixBrowse as a member of undocumented symbol $.fn

There is a simple way to tell this to it. Just write these comments at the top of the file where you would like to use jQuery.

/**
 * See (https://jquery.com/).
 * @name jQuery
 * @class
 * See the jQuery Library  (https://jquery.com/) for full details.  This just
 * documents the function and classes that are added to jQuery by this plug-in.
 */

/**
 * See (https://jquery.com/)
 * @name fn
 * @class
 * See the jQuery Library  (https://jquery.com/) for full details.  This just
 * documents the function and classes that are added to jQuery by this plug-in.
 * @memberOf jQuery
 */

This tells jsdoc about jQuery and jQuery.fn and you can then begin writing documentation for your plugin.

/**
   * myAwesomePlugin - an awesome jQuery plugin.
   *
   * @class myAwesomePlugin
   * @memberOf jQuery.fn
   */
  jQuery.fn.myAwesomePlugin = function() {
  }

You can also write a file overview to be included in the documentation by writing this at the top.

/**
 * @fileOverview Contains the awesome plug-in code.
 * @version 0.1
 * @author Pulkit Goyal <pulkit110@gmail.com>
*/

The detailed documentation for functions is very similar to JavaDoc style. Here’s a sample.

/**
 * Get user defined options from data-* elements.
 * @param {jQuery object} elem - the element to retrieve the user options from.
 * @returns {Object} options - User's options for the plugin.
 */
function getUserOptions(elem) {
}

HackerNews in 3D

Mon, 09 Apr 2012 21:22:14 GMT

Today I noticed the 3D Inspect element feature for Firefox.

Here’s how you can get it: Step 1: Make sure you have the latest version (at least 11.0). Get the latest version here.

Step 2: Right click anywhere in the page then click Inspect Element (Q).

Step 3: At the bottom of the page click the 3D (M) button.

Its a great way to show off Firefox’s 3D capabilities. But I haven’t been able to find where this can be useful. Don’t forget to leave comments about what you think of this feature and where it can be useful as compared to the traditional inspect element.

Similarity Metrics on Twitter

Fri, 16 Mar 2012 23:01:03 GMT

Most of the researches in the field of community detection consider only social connections as the similarity measure for obtaining communities in the network. In this post (as a part of my series of posts on Data Mining and Analysis on Twitter), we will discuss about various other possible similarity measures between different users and discuss why and how they can be used to cluster users in the network.

User Connections: This is the similarity measure that is used the most in the literature to define a connection between two users. We define a social connection on Twitter to be a following or a being followed relationship between two users on twitter. As we will see in further sections, this is one of the most dominating factors that produce a community structure on twitter and this is the reason that it has been used so extensively in most of the researches. We define an edge of weight between two users and if either follows or follows . Therefore, the users social connections (or the user connections matrix as we will call it now) is a symmetric matrix with a link between two users who have either of the following or being followed relation between them.
User Mentions: This is another form of a connection that can be defined between two users. As described before, mention is the event of mentioning another user in our tweet. The image below shows an example of a mention on twitter. The user named @EPFLNews has mentioned another user named @SmallRivers. This mention is as a result of @EPFLNews saying something about the user @SmallRivers and therefore wanted to let him know. It has been observed that a mention occurs as a result of discussion or good relationship between users and therefore, it can serve as a good measure of relationship between users. Another important motivation to consider mention as a similarity measure is that it corresponds closely to user connections but is much more selective. We count the number of mentions that two users make on twitter and assign the weight of the link between two users and as the total number of tweets posted by that mention and the tweets posted by that mention the user .

An example tweet showing a mention on twitter

Description Content Similarity: Users on twitter can post a description about themselves which is shown in their profile. The image below shows an example of description on twitter for the user with screen name @pulkit110. This description can sometimes be used to measure the similarity between users on twitter. Since this description generally describes the keywords about what the user likes to do or where he works/studies at, it can serve as a very good suggestion of user similarity on twitter. Therefore, we consider the cosine similarity between the users’ descriptions as one of the similarity measures in determining the clusters of users on twitter.

Example of user description on Twitter

Tweet Content Similarity: The most popular concept of twitter is the concept of tweets. It is tweets that most users on twitter are interested in and therefore, it can be used as a similarity measure between different users. We define the tweet similarity between two users as the cosine similarity between the documents formed by combining the tweets of a user into one. The text similarity measure between the tweets helps us to observe if the users are interested in talking about similar topics. If the users talk about the same topic then it is quite possible that they are interested in similar things and is an indication of good similarity between them.
Hash tag similarity between users: Hashtag is a unique concept on twitter which allows users to specify important keywords in their tweets my prefixing ‘#’ before a keyword in a tweet. Hashtags have been used on twitter to set trending topics as well as start chat rooms etc. The hash tags allow users to specify what they think as an important keyword in their tweet and therefore can be considered as a very strong factor to compare two users’ similarity. The image below shows an example of the user @diwakarsapan posting a tweet with hashtag #IEUsers. The hashtag shows that the user wants to emphasize on a particular keyword in his tweet. We define the hash tag similarity between two users as the cosine similarity between the collections of hashtags of the different users.

Example of hash tag on twitter

All posts in this series:

Analysis of Spectral Clustering on Twitter

Fri, 16 Mar 2012 21:02:38 GMT

I recently worked on Data Mining and Analysis on Twitter as my semester project at EPFL. During the 4 months that we worked on the project, we were able to achieve many goals. In this post, I will be describing how we can cluster a small group of users on Twitter based on different similarity metrics and a brief comparison of several clustering methods that could be used for this.

For this analysis, we used three different user lists on twitter as the ground truth data for a group of about five hundred users. We obtained all the tweets from the users who were listed in the three lists and then tried to obtain clusters by using different similarity metrics using the spectral clustering algorithm. In addition to this, we also explore different connections between users in addition to just the social connections in order to find out other features that affect the users being listed together. We present results of applying spectral clustering algorithm using the modularity matrix and the symmetric normalized Laplacian matrix. We compare the results of these approaches while using several different input matrices formed by different combination of the above similarity measures.

If we have a set of n objects $$x_1,x_2,x_3,\dots,x_n$$ with a pairwise similarity function defined between them which are symmetric and non-negative. Spectral clustering is the set of methods and techniques that partition the set into clusters by using the eigenvectors of matrices.

The motivation behind using eigenvectors for clustering is that the change of representation induced by the eigenvectors makes the cluster properties of the initial data set much more evident. In this way, spectral clustering is able to separate data points that could not be resolved by applying directly k-means clustering, for instance, as the latter tends to deliver convex sets of points. Since the introduction of spectral methods in "Lower Bounds for the Partitioning of Graphs" there have been several researches where scientists have tried using different matrices for the calculation of eigenvectors followed by applying clustering on the eigenvectors. A complete discussion about the different spectral algorithms and matrices would take a new post, so I wouldn't go into much details here.

Now, I will present the results of applying spectral clustering algorithm using the modularity matrix and the symmetric normalized Laplacian matrix. We compared the results of these approaches while using several different input matrices formed by different combination of the above similarity measures. Before I begin, let's have a look at the spy plot of the user connections. The users have been ordered by the lists that they belong to and therefore, we can immediately observe three communities present in the network by looking at the plot.

We present the results of application of the algorithm on users’ social connections as well as several other individual similarity measures (user mention similarity, description content similarity and tweet content similarity) followed by a simple combination of the different similarity measures. For finding the combined similarity measure, we sum all the different similarity measures. Since the different similarity measures can be on different scales, these similarity measures are normalized before we add them together and apply the clustering algorithms. Therefore, the adjacency matrix that corresponds to the combined similarity measures is the sum of all the individual normalized adjacency matrices.

In order to measure the accuracy of our clustering algorithms, we use several different cluster evaluation objective functions to compare obtained clusters with the ground truth data of clusters which represents the distribution of the users into different lists. I will present the values of Normalized Mutual Information and RandIndex during the analysis. A higher value of these factors means a better correspondence with ground truth. The table at the end of this post summarizes these values for different clustering algorithms and similarity metrics.

I will now show some visualizations of our obtained results. The results have been obtained using Gephi for visualization. The communities are represented by different colours in the visualizations. Note that the arrangement of the nodes in the space doesn’t represent any communities. The arrangement of nodes in the visualizations corresponds to a layout. We keep the same layout for all the visualizations so that it is easy to see the results of the clustering process. The different clusters in the visualizations are represented by different colours. This means that all the nodes in the visualization that have the same colour have been placed into one cluster for that visualization. Figure (a) shows the communities formed using the ground truth data. Figure (b) shows the clusters obtained for the network using the user connections as the only similarity measure using the modularity matrix for spectral clustering. Figure (c) shows the results of community detection using the combination of all similarity measures for the modularity matrix. Finally Figure (d) shows the results for community detection applied on combined similarity measures using the Symmetric Normalized Laplacian matrix.

a b

c d

We can observe that user’s social connections are the most dominating factors for this group of users while dividing the users into different communities. We can also observe that the high values of benchmark results correspond to the good community detection that corresponds closely to the ground truth data by looking at the visualizations in Figure (a) and Figure (b) which show the ground truth clusters and the clusters obtained by applying community detection using modularity matrix on social connections respectively. The other individual similarity measures like the user mentions, tweet content similarity and description content similarity don’t perform very well when used for community detection using the modularity matrix. We also observe that the results for combined similarity measures, i.e. the sum of connections, mentions, tweet content similarity and description content similarity doesn’t perform as well as the connections. This means that the addition of low information contents like the mentions, tweet content similarity and description content similarity decreases the accuracy of the clustering algorithm even in the presence of the highly informative social connections. A reason for the bad performance of the similarity measures based on the tweets, descriptions and mentions can be that the group of users are similar and generally post similar content on the web. This also means that the user behaviours don’t seem to be consistent with the ground truth data.

However, the detection of communities for the user’s social connections using the Symmetric Normalized Laplacian matrix fails. This is because the Laplacian based methods are known to be quite sensitive to the presence of disconnected nodes in the graph. Therefore, the results of the social connections for the Symmetric Normalized Laplacian are similar to the other individual results which mean that we are not able to reconstruct any valuable cluster information when using the Normalized Laplacian. But the combined matrix performs consistently even when using the Normalized Laplacian for community detection. This is because addition of several different kinds of information to the social connections makes it a connected graph and therefore, we now observe results consistent with the ones obtained while using modularity matrix for community detection.

Therefore, we can conclude from these results that user connections are a very good measure of information and for this model; they give the best clustering results when used with the modularity matrix for spectral clustering. We can also see that adding several other non-informative measures to this layer decreases the accuracy considerably when used with the modularity matrix. However, an interesting observation occurs from our discussion and results regarding the Symmetric Normalized Laplacian matrix that even a highly informative layer (user connections) can prove to be a very bad indication of clustering if not used with the correct clustering algorithm due to the presence of disconnected nodes in the graph. We also found that even adding a non-informative (or slightly informative) layer as in this case can improve the connectivity and therefore improve the clustering results.

Similarity Matrix	Modularity Matrix		Symmetric Normalized Laplacian Matrix
Similarity Matrix	NMI	RI	NMI	RI
User Connections	0.3868	0.7174	0.0077	0.3374
Mention	0.0130	0.3398	0.0077	0.3374
Tweet content similarity	0.0074	0.3371	0.0077	0.3374
Description content Similarity	0.0780	0.5254	0.0088	0.3381
All combined	0.2500	0.6175	0.2931	0.6472

All posts in this series:

File download using HTTP request

Tue, 14 Feb 2012 15:16:01 GMT

Downloading a file from a server is trivial and so is the implantation of it using httpservletresponse. But after fighting with for a long time I finally came across this jQuery plugin. This works well as long as we make HTTP request using GET or POST to server and the server responds with a file to download.

How to use this

For this you need to download the plugin and include the file- ```javascript ```

The plugin accepts 3 arguments for url, data, and method. You can pass data to these arguments just as you would to jQuery's $.post or $.get functions, and assuming the server has no problems handling the request, the front end will respond with a prompt for a file download and the user never needs to leave the page. Here's an example call to the plugin: ```javascript // the second argument can't be left blank, so if you don't need it, you can write anything $.download('get', 'file=file', 'get'); ```

and you’re done!

Of course, if the server doesn’t send the correct response, it wouldn’t work. These are the list of headers that I used for the response sent from the server.

// get your file as InputStream
InputStream is = ...;
// copy it to response's OutputStream
IOUtils.copy(is, response.getOutputStream());
response.flushBuffer();

// set headers
response.setContentType("application/pdf");  // Your content type
response.setHeader("Content-Disposition", "attachment; filename=somefile.pdf");

Using Hibernate to filter results by Date

Sun, 22 Jan 2012 14:28:56 GMT

Today, I was trying to access some data for my project on Twitter and filtering it using the creation date of the tweets. So I began by creating the usual Criteria and added a few Restrictions. For filtering by date, I created Date objects for startDate and endDate using Calendar.getTime() method and added the following Restriction, c.add(Restrictions.between("createdAt", startDate, endDate); Surprisingly, this didn’t work as I had expected and I was getting an empty result. I Googled for it but couldn’t find much. The only thing was that the Restriction should work given the date. Well, I was in a hurry and didn’t try to dig it through. As a workaround, I tried writing HQL query to get the results but that didn’t work either.

This is what worked. I created the date using SimpleDateFormat and parsing the date. Here’s what I did:

SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd hh:mm:ss");
Date startDate = format.parse("2011-11-09 00:00:00");
Date endDate = format.parse("2011-11-017 00:00:00");

c.add(Restrictions.between("createdAt", startDate, endDate));	

List<StatusDto> statuses = c1.list(); // It works fine now

Its a quick hack and did work out for me. I was in bit of a hurry and didn’t try to dig up the reason for it now working earlier. Maybe, someone can point my mistake out in the comment. Anyway, I will update the post if I find the reason for it.

Stack Overflow like keyboard buttons in post

Mon, 02 Jan 2012 23:01:46 GMT

Here's a quick hack to get yourself one of those Stack Overflow like keyboard buttons. For those of you who don't know what these are, have a look here. I am writing this blog post while I wait for the India Australia cricket match to begin which is in a few minutes from now, so I'll be quick :).

To add these cool looking keys in your post, just write them in <kbd>...</kbd> tags. Simple, right? Not quite. We’re not done yet. We need to do just one more thing. Add the following in your stylesheet:

And we are done. Give it a try. These look wonderful to me. Here's an example: Ctrl + D

kbd {
	padding: 2px 4px;
	white-space: nowrap;
	color: black;
	background: #EEE;
	border-width: 1px 3px 3px 1px;
	border-style: solid;
	border-color: #CCC #AAA #888 #BBB;
}

Here's a small keyboard built using these buttons :)

~ 1 2 3 4 5 6 7 8 9 0 - = Backspace

Tab q w e r t y u i o p [ ] ENTER

Caps a s d f g h j k l ; ´ +

Shift < z x c v b n m , . / Shift

Ctrl Win Alt Space Alt Win Menu Ctrl

Testing your web project using jqUnit

Fri, 30 Dec 2011 23:48:48 GMT

I have been wanting to write this post since a long time. Finally, here it comes.

This is something that everyone working in a community should know. Testing! I didn’t consider it important enough until Michelle explained its importance to me during the Image Editor project for Google Summer of Code. Now that I started using it, its become so important that its hard to imagine a project with a team without test cases. And the most important thing is writing these tests is amazingly simple and it saves you a lot of time in the future.

I will put up the importance very briefly here. When we develop anything in a team with more than one developer working on the same project, there may be some changes that we make that might break something that others had previously written because they had assumed something which wasn’t very obvious to us. Having tests ready for the project comes to our rescue here. If the tests pass, we can lie in peace knowing that the changes that we have made don’t break anything.

As I mentioned earlier, writing test-cases is amazingly simple using all those frameworks out there. I will be describing the process of writing tests for jQuery using jqUnit. Before, we start writing the test, this is how we can initialize a new test case:

var myTests = new jqUnit.TestCase("My Application Tests");

We can now start writing our tests:

// 1
myTests.test("Test Scenario 1", function () {
	// Write some tests here
});

Now lets start writing some simple tests. Don’t worry. Writing unit tests requires us to just call a few functions that we wrote and check it against a return value that we expect. This is something that you always think of when writing the function, so writing the test is just a little extra effort which certainly pays off in the future. Finally, here’s how to write the tests:

jqUnit.isVisible("My component is visible", "#my-component-id");
jqUnit.notVisible("My hidden component is not visible", "#my-hidden-component-id");
jqUnit.assertNotNull("My function 1 doesn't return null", myApplicationInstance.myFunction1());
jqUnit.assertNotUndefined("My function 1 doesn't return undefined value", myApplicationInstance.myFunction1());

jqUnit.assertFalse("My function 2 returns false", myApplicationInstance.myFunction2());
jqUnit.assertTrue("My variable 1 is set", myApplicationInstance.myVar1);

jqUnit.assertEquals("My function 3 returns correct value", "expectedValue", myApplicationInstance.myFunction3());

The tests are pretty obvious. This takes care of the actual tests. Now we need something where we can see the results of the tests. Lets make an html file where we can include the js file that we just created.

<!--  This is the QUnit test css file -->
<link rel="stylesheet" media="screen" href="lib/qunit/css/qunit.css" />

<!--  These are the jqUnit test js files -->
<script type="text/javascript" src="lib/qunit/js/qunit.js"></script>
<script type="text/javascript" src="lib/test-core/jqUnit/js/jqUnit.js"></script>

<!-- The test file that you just wrote -->
<script type="text/javascript" src="../js/myTests.js"></script>

<!-- Other imports, like jQuery and your application imports -->

And finally the markup to render the tests:

<body id="body">
    <h1 id="qunit-header">My Application Basic Tests</h1>
    <h2 id="qunit-banner"></h2>
    <div id="qunit-testrunner-toolbar"></div>
    <h2 id="qunit-userAgent"></h2>
    <ol id="qunit-tests"></ol>
</body>

If you want to have a look at the complete test files for a live project, see my tests for the Fluid Infusion Image Editor here.

A summer with Fluid and Google!

Sun, 06 Nov 2011 00:53:42 GMT

With the end of Google Summer of Code 2011, I finished my summer with a great experience. I got the opportunity to interact with some of the best developers in the field of open source and got to learn a lot about web development, open source software and project management as I finished developing a new component, Image Editor for the fluid framework. One of the most important things that I learned while working with them was the importance of building tests. To explain more about it, I would be writing a blog post soon about jqunit soon. You can have a look at an online demo for the component here.

Looks cool? Integrating it into your own webpage is really easy. Checkout the tutorial and docs. With the Image Editor component you get two more components, the TaggerUI and CropperUI. But nothing stops you from exploring more about these separately. In fact, these components come prepared to be used individually for your website. For more details about all the component tutorials and API docs related to Image Editor, Tagger and Cropper, visit the fluid wiki. For the web developers out there, have a look at the code of the components on github.

In addition to the components that I created, the fluid community has a lot to offer. I enjoyed working with them and now enjoy using the fluid components. For a detailed demo of all the fluid components, visit the fluid demonstration portal.

Overall, it was one of the best summers that I could have had. I hope to work with Fluid and Google again.

How to update Android SDK to 4.0

Sat, 29 Oct 2011 12:33:26 GMT

Google has recently launched android 4.0 and it is no doubt one of the best updates so far. For more details about Android 4.0, see here. To update your android sdk to version 4.0 on Windows, browse to your android installation directory (C:/Program Files (x86)/Android by default on 64bit Windows). Start SDK Manager.exe as an administrator (from the right click menu). It should then automatically search for updates and ask you for installation. Begin the installation. So far so good. Now, something strange happens: “the latest update failed to install because it couldn’t rename the tools folder in android-sdk-windows”. You might give it a try by trying to restart the update after quitting windows explorer and disabling your antivirus software for a while. But it didn’t work for me and might not work for you as well. The problem here was that the SDK Manager itself was using the directory and therefore wasn’t able to delete it and replace it with a new one. Here’s a work around for the problem that worked for me: Solution:

Make a copy of the tools folder itself (keeping it at the same directory tree level, thus "tools" and "tools-copy" were both in the "android-sdk-windows" folder).
Run Android.bat from that copy (I recommend running it as an administrator)
Re-run the update (it should now update the original, not-being-used-at-the-moment tools folder, among whatever other items it needed to).
Close the SDK, delete the folder that you built in first step (Optional) (You might need to kill the adb.exe process first - not sure why that always persists but you can't delete the folder without doing that).
Restart the SDK from the normal (now-updated) tools folder.

Now you can work with the updated sdk just like you did with the old one. For a good tutorial on how to start developing apps, visit here.

You might also run into a problem when you now run eclipse for developing android applications. The error that I received was this:

This Android SDK requires Android Developer Toolkit version 14.0.0 or above. Current version is 12.0.0.v201106281929-138431. Please update ADT to the latest version.

To solve this problem:

Go to Help → Install New Software.
On Work with: type https://dl-ssl.google.com/android/eclipse/ and press ENTER.
Wait for Eclipse to fetch the repository. An item named Developer tools will appear in the list. Mark it for install, press Next and follow the steps to install the ADT tools.
When finished, it will ask to restart Eclipse. Make sure you do this.
When Eclipse restarts, all your Android SDK packages should show up again.

[How to]Integrating Maven with Eclipse

Mon, 17 Oct 2011 01:26:39 GMT

If you are coming to this post this means you know enough about Eclispe and Maven. For those who don't know about Maven, (from maven.apache.org) Maven, a Yiddish word meaning accumulator of knowledge, was originally started as an attempt to simplify the build processes.

Now a days most of the projects build through Maven. I use Maven2Eclispe (m2e) plug-in for using maven with Eclispe. To add this to your Eclipse follow these simple steps-

Go to Help(in Eclispe) click on Install New Software. Add this link https://m2eclipse.sonatype.org/sites/m2e in work with option. Tick Maven Integration for Eclipse from the check-box given below.
Don’t forget to tick the Contact all update sites during install to find required software
Click finish to complete the installation.

You might get an error saying

Cannot complete the install because one or more required items could not be found. Software being installed: Maven Integration for Eclipse (Required) 0.12.0.20101115-1102 (org.maven.ide.eclipse.feature.feature.group 0.12.0.20101115-1102) Missing requirement: Maven Integration for Eclipse (Required) 0.12.0.20101115-1102 (org.maven.ide.eclipse.feature.feature.group 0.12.0.20101115-1102) requires ‘org.eclipse.emf.ecore.edit 0.0.0’ but it could not be found.

This is due to it requires Eclispe Modelling Framework (EMF) for its installation but ir comes from different update site than M2Eclispe and therefore you need to add EMF framework too. To do so add download.eclipse.org/releases/helios site also and tick EMF for installation, click finish and you are DONE with it.

Using Twitter4j with SSL connection

Sat, 01 Oct 2011 14:29:12 GMT

This morning my Twitter Data Collection project suddenly stopped working. Suddenly, it failed to connect to the Twitter servers. It is as a result of the recent announcement made by twitter that it would allow only SSL connections starting from 29.09.2011.

I had been using twitter4j to connect to twitter stream. Even after setting the twitter4j.http.useSSL property to true, it failed to connect.The apparent solution to this was to update the twitter4j API to version 2.2.5-SNAPSHOT as announced through this tweet by twitter4j. But even after updating my project’s pom.xml to the new version, the problem wasn’t resolved. I figured out that the SNAPSHOT version wasn’t getting downloaded for twitter4j-core. Here’s how I fixed the problem:

<repository>
	<id>twitter4j.org</id>
	<name>twitter4j.org Repository</name>
	<url>https://twitter4j.org/maven2</url>
	<releases>
		<enabled>false</enabled>
	</releases>
	<snapshots>
		<enabled>true</enabled>
	</snapshots>
</repository>

This fixed the problem. Finally my data collection is back on track! :)

Firefox Takeout: Take your Firefox data with you

Wed, 07 Sep 2011 10:13:53 GMT

This post comes after a long time as I was busy in traveling. This summer I was working as Google Summer of Code Student’s for Fluid. After completing it successfully, when I was leaving the PC at my office on which I was working for my project I realized that I will lose track of all the technologies, sites and blogs etc. I had bookmarked as well as my browsing history, which Firefox uses to show suggestion while user types something in address bar. On googling a bit about this, I found out a solution for this. But there wasn’t any software available to assist me with the process. So I decided to build one for me so that I wouldn’t have to search again whenever I move my system. I then decided to build a software by which I would be able to see all suggestions from previous system as well as suggestions which was already present in the second system, and named it Firefox Takeout which can be found at here. This software works only on those Windows system on which English as Language is installed.

So, what does the software actually do? This software works in 2 steps. First screen provides user to choose from 2 options as shown-

When user chooses 1st option that is Takeout Firefox data from first system; file used by Firefox for showing suggestion to the user while typing, is copied inside Firefox Transform folder located on desktop (It creates the folder if it doesn't exist). This completes 1st step. Now user can copy this folder and paste it on desktop of 2nd system. This is pre-requisite for the 2nd step. Now on the 2nd system, user can select 2nd option, i.e., Bring in Firefox data, this will send the file to the destination.

This software was not difficult to build but it simplifies the life of the user to a great extent when user switches from one system to another specially to those uses which uses bookmarks and are not able to keep track of site visited by them. I think you will enjoy using this software. I will be improving the software's functions, look and feel in the future. Feel free to follow the development on the project page at sourceforge. If you face any problems, I would be glad to help.

Javascript Image Manipulation using HTML5 canvas element [Tutorial]

Sun, 12 Jun 2011 23:20:04 GMT

I have been working with HTML5 and Javascript for my Google summer of code project for the Fluid Project. This post is the first of a series of tutorials which I would be posting related to Javascript, HTML5 (especially canvas element) and Fluid Infusion.

I have been exploring ways to get image pixels from an image object using Javascript. As far as I know, the only way of doing this so far is by using the HTML5 canvas element. So, you can either build a canvas element yourself and get the pixels or use some nice Image Processing library such as Pixastic. I would now be describing more about image processing using canvas element. I assume that you know a bit about the HTML5 canvas element beforehand. For knowing more about canvas, I recommend going through these tutorials by Mozilla. I also assume that you have got an Image object for which you want to manipulate pixels.

Top obtain the pixels for a given image, you first have to draw that image on canvas and then get the pixels using the getImageData method on canvas context.

var imageManipulationCanvas = document.createElement('canvas');
imageManipulationCanvas.height = image.height;
imageManipulationCanvas.width = image.width;
var imageManipulationCtx = imageManipulationCanvas.getContext('2d');
imageManipulationCtx.drawImage(image, 0, 0); // Draw image on temporary canvas
var myImageData = imageManipulationCtx.getImageData(0, 0, image.width, image.height); // Parameters are left, top, width and height

The image data object has three fields: width, height and data where data is the CanvasPixelArray object which contains pixel data for image.

The CanvasPixelArray object can be accessed to look at the raw pixel data; each pixel is represented by four one-byte values (red, green, blue, and alpha, in that order; that is, “RGBA” format). Each color component is represented by an integer between 0 and 255. Each component is assigned a consecutive index within the array, with the top left pixel’s red component being at index 0 within the array. Pixels then proceed from left to right, then downward, throughout the array.

The CanvasPixelArray contains height x width x 4 bytes of data, with index values ranging from 0 to (height x width x 4)-1.

Now you can manipulate the pixels as you want for any image. However for simple operations like resizing and cropping, I prefer using different variations of the drawImage function rather than manipulating the pixels myself. Here’s how to resize an image using drawImage:

var imageManipulationCanvas = document.createElement('canvas');
imageManipulationCanvas.width = newW;
imageManipulationCanvas.height = newH;

var imageManipulationCtx = imageManipulationCanvas.getContext('2d');
imageManipulationCtx.drawImage(that.image, 0, 0, resizeW, resizeH); // Draw resized image on temporary canvas
var resizedImageDataURL = imageManipulationCanvas.toDataURL(); //get DataURL for cropped image
return resizedImageDataURL; //DataURL can be directly used to make new image pbject by using image.src

And, the following for cropping an image:

var imageManipulationCanvas = document.createElement('canvas');
imageManipulationCanvas.height = newH;
imageManipulationCanvas.width = newW;
var imageManipulationCtx = imageManipulationCanvas.getContext('2d');

//use drawImage(Object image, float sx, float sy, float sw, float sh, float dx, float dy, float dw, float dh)
imageManipulationCtx.drawImage(image, croppingDimensionX, croppingDimensionY, croppingDimensionW, croppingDimensionH, 0, 0, croppingDimensionW, croppingDimensionH); // Draw cropped image on temporary canvas
var croppedImageDataURL = imageManipulationCanvas.toDataURL(); //get DataURL for cropped image
return croppedImageDataURL;

Tutorial 4: Building the Yellow Pages Application

Sun, 05 Jun 2011 21:51:41 GMT

Go through Tutorial 3 first.

In this tutorial, we would build a Yellow Pages Application. We will continue it from the last tutorial and extend it to have one more form and another dialog, the MessageQueryDialog to display the required data. We will also look into Symbian File Handling to retrieve the data about businesses.

First, add another form in the resource file in a similar fashion as the earlier form(define a resource in resource file and class to handle the form events) to provide for selection of category. Go through the previous tutorial if you have any problems in adding another form. Once the form is added, we need to execute the form after the city is successfuly selected in the first form. To do this, we make a new form object in the SaveFormDataL() method of the first form(just like we did in HandleCommand for the first form). Once, we have extracted the city and categories, we will now extract the corresponding information about the businesses in the city for the category from te file that we have stored in the phone’s memory.(You need to make those files(or copy them from the YPages folder) and place them in the memory) For running the application on the emulator, just place the folder YPages from the current directory to the SDK’s winscw folder-

Default Path: C:S60devicesS60_5th_Edition_SDK_v1.0epoc32winscwd

File Handling in Symbian:

First, you need to obtain a handle to File Server session and connect to it. You can do this by:

RFs fileServer; User :: LeaveIfError (fileServer.Connect());

Now, we will use the RFile class to read data. Open the file for reading. The EFileStreamText flag denotes that the file opened is a text file. The EOF is indicated by the return value of TTextFile::Read(), which is KErrEof.

RFile file; User::LeaveIfError(file.Open(fileServer, KMyTextFile, EFileRead|EFileStreamText));

The class to read a text file is TFileText. It is declared in f32file.h header file and a library named efsrv.lib(add this to the mmp file). Before using TFileText, we have to open a file using RFile because TFileText does not have an Open method. TTextFile has Read() method which requires a 16 bit descriptor parameter because Symbian OS stores text files in unicode format. Do not use TFileText to open a file other than unicode. we will get garbage characters if trying to access non Unicode files, such as ASCII files. Go through the source code to understand clearly.

Close the file server session after the reading is completed.

After successfuly reading data from the file, we will now print the read data in a MessageQueryDialog. We must first define the resoruce for this dialog. We have defined a raboutheading_pane resource in the resource file. It is similar to previous dialogs, define the flags, cba function, and items. the difference here is that we do not define a for for this dialog. Instead we define 2 items, heading pane, and a CAknMessageQuery to hold the text.

After defining the resource in the resource file, we must execute the resource in the SaveFormDataL() after reading the data from file. For this, create a new object of type CAknMessageQueryDialog and Prepare it with the resource defined in the resource file. Then, set the header text and finally call RunLD() on dialog to display it. The text to be displayed in the dialog was passed alongwith the dialog’s constructor while making the new object.

Download Source code for this tutorial here.

Tutorial 3: Creating Forms & Supporting Multiple Languages

Sun, 05 Jun 2011 21:28:24 GMT

Go through tutorial 2 first.

In this tutorial, we look into more complex dialogs. For our application, we use the form dialog. A form displays a set of data fields in the form of a list, with each data field in the list consisting of a label and a control. The label can be on the same line as the control, or it can be on a separate line, with the control below it. In addition, a form dialog is automatically associated with a standard menu that supplies the options:
Add field, Edit label, Delete field, Save and, optionally, Edit.

Selecting one of the first four of these options results in a call to the appropriate one of the CAknForm functions: AddItemL(), EditCurrentLabel(), DeleteCurrentItem() and SaveFormDataL(). In our application, we use only the save option which in turn calls the SaveFormDataL() function.
An S60 form has two modes: in 'view' mode it acts as an application view that displays a list of data items, and in 'edit' mode it can be used to modify the data items displayed. By default, it starts up in 'view' mode and you can switch to 'edit' mode by selecting the Edit menu option. When you have finished editing the data, you press the right softkey (temporarily labeled Done)to return to the 'view' mode. A form is actually more powerful than a dialog. If, for example, the data items it is displaying are the fields of a database record, you can implement the commands described above to add, delete, or modify entire records.
You can specify that the form should be edit-only (via a fiag in the FORM resource), so that the form is always in 'edit' mode, and in this case, the Edit menu option does not appear.

You can also override its DynInitMenuPaneL() to disable some or all of the other menu options. You specify a form in the resource file by creating a FORM resource and assigning it to the form attribute of a DIALOG resource (or a PAGE resource for multipage dialogs). The FORM resource contains the list of DLG LINES that specify the label and control for each field in the form's list.

We can also provide support for multiple languages in the application through the rls file. While you can put text strings directly within the resource file, this is not recommended if you need your application to support different languages. Symbian recommends that you put all your strings into a RLS file, and then include this string file using #includein your RSS file. There should be a separate RLS file for each language you support.
Each string in the RLS file is defined using the rls string keyword. For example: rls_string STRING_r_gui_start "Start";

As you can see, the DIALOG resource defines the flags and softkeys, and the form attribute points to a FORM resource. This resource specifies the dialog's content which, in this case, consists of one dialog line: a S60-specific control, known as a pop-up field (type EAknCtPopupFieldText and control structure POPUP FIELD), which is used to select the city. The FORM resource has an additional flags attribute, which is used here to set each control and its prompt to be displayed on separate lines, and to set the 'edit only' mode that was mentioned earlier. We also define the list of cities to appear in the popup field using a RESOURCE ARRAY r_city_list.

PreLayoutDynInit() is overridden to set the initial values of the controls in the form. We also override the SaveFormDataL() method to perform the save operations. For our application, we obtain the index of the selected city from the popup menu using the CurrentValueIndex() method in popupFieldText. we then print a dialog similar to the one printed in the second app to display the selected city index. The descriptor method Format() does this. The format string supplied to Format() is very similar to the format string in C, supporting %d, %s, %f, etc.

This wont display the form on the screen. To make the form appear on the screen once we select the start menu item, we need to handle the command in HandleCommand() method in Application Ui Class. We create a new form object and execute R_THIRDAPP_DIALOG which was defined in the resource file.

Download Source code for this tutorial here. Go through tutorial 4 now.

Tutorial 2 : First Dialog

Sun, 05 Jun 2011 21:14:45 GMT

Go through tutorial 1 first.

Now that you know how to print text on the screen, lets go to a little more complex application. We aregoing to add a menu to the application, how to handle commands in a little more detail, and finally create an alert dialog.

Here are the changes that it makes to the first tutorial.

Add the menu bar resource in the resource file to display the menu. The menubar attribute of EIK APP INFO is assigned a resource of type MENU BAR, which specifies the application’s default menu. Menu bar resources have one or more menu titles (type MENU TITLE)and each menu title points to a menu pane (type MENU PANE). The menu bar in the example, r_SecondApp_menubar, has a single menu title and this points to menu pane r_SecondApp_menu. Menu panes define the actual menu items (type MENU ITEM), whichthe user selects to invoke some operation in the application. r_SecondApp_menu defines a menu item labeled ‘Start’ that sends the command ESecondAppCommand to the GUI command handler code when the user selects it (so the code can display the Application’s dialog).
Now, we need to define the command codes used in the resource file. These are defined in the .hrh file. The file SecondApp.hrh contains the command values that the controls send (specified in the resource file) for the application code to handle. In this case we have only one, used when Start is selected: ``` enum TSecondAppIds { ESecondAppCommand = 1 }; ```
the menu item ‘Start’ is selected, the GUI framework invokes the HandleCommandL() method, passing it the command ESecondAppCommand (the command specified in the menu resource in the SecondApp.rss file). HandleCommandL() responds to this command by popping up an alert window with the message ‘Your First Dialog!’.We can use the iEikonEnv->AlertWin() function for the pop-up since this is a core GUI method available to all platforms. This function takes one arguement which is a descriptor.
- Descriptors are classes that represent data buffers and allow you to safely access them. Symbian OS uses descriptors to store and manipulate strings (as opposed to NULL-terminated C strings), as well as to manage binary data. Descriptor classes, although containing many features, are optimized for minimal overhead, since they are designed to run on memory-constrained devices. There are multiple descriptor classes available in Symbian. You’ll need to use descriptors to call many of the Symbian OS API functions.
- A String Literal(Descriptor) in Symbian is declared as:
  [cpp] _LIT(KMessage,"My First Dialog!");[/cpp]
  The LIT macro is called to take the string "My First Dialog!" and stores both the string and the string’s size in the descriptor literal KMessage.

Download Source code for this tutorial here. Go through tutorial 3 now.

Tutorial 1 : Learning the basics

Sun, 05 Jun 2011 20:53:11 GMT

Now that you have downloaded the SDK and know how to make projects in Carbide, we will now look into our first GUI Application which prints a line of text in the center of the screen. The main goal of this tutorial is to get a feel for developing a basic application by actually building and running one on the emulator.

Application Architecture

A minimal Symbian GUI Application must contain the following 4 classes:

Application View: The view class implements the application's screen display, including drawing the window and the creation of the initial screen controls. The class is inherited from CCoeControl which is fine for Applications with just one view class. We would only be discussing a single view class for now.

Application UI: This class instantiates the application view and handles the commands sent from the application's GUI controls.

Application document: This class handles the non-GUI data aspects of the application – the application data. It also instantiates the application's UI class.

Application: This class is used to identify the application (by returning the application's UID) and to instantiate, and return a pointer to, your application's document class.

Now let us discuss these classes one by one.

Application View Class:

The application view class handles the presentation of your application on the smartphone's screen, as well as allowing the user to interact with your program. In Symbian OS all objects drawn to a screen are controls – including the application view, which is a custom control. It has the following methods:

NewL, NewLC and ConstructL: These methods construct a new object of the view class. We will discuss about why we took these 3 methods and not just the constructor to construct the new object later in error handling.

Draw(): Draw() is a method called by the framework for every control in order to draw it to the screen. The application view is a control, and, we implement the Draw() function to output the text 'First Application' in the center of the window. The drawing is performed by opening a graphics context (GC), getting a font, and calling the context's DrawText() function. Cleanup is performed on the font upon

completion.(Cleanup will be explained in error handling.)

Application UI Class:

Your application's UI class is responsible, upon construction, for creating the application's default view. In S60, for basic one-view applications (i.e., the view is a simple control), the UI class contains the command handler that handles all the commands the users initiate via the GUI. If you are using multiple views, however, this command handler would be in the view class. It has the following methods:

ConstructL(): This function does the work of construction of an object of View Class.
HandleCommandL(): This function handles the commands passed from the Application's GUI controls. In this application, we just need to handle the commands for exiting the application.

Application Document Class:

The document class has two purposes, the first of which is to represent the application's persistent data(Data that needs to remain after application is exited). The other is to create the application UI instance in which this data (if any) can be manipulated. For applications that have no persistent data, and therefore are not file-based, the document class simply implements the CreateAppUiL() function to return the application's UI object.

Application Class:

This is the first thing the GUI framework creates. It has the following methods:
CreateDocumentL(): This function creates an object of Document class and returns a pointer to it.
AppDllUid(): This function returns the Application's Uid.

E32Main()Entrypoint and NewApplication()

A Symbian OS GUI application is a process executable (EXE file) and therefore must contain an E32Main() entrypoint function. For a GUI application this just calls EikStart::RunApplication(), passing it a pointer to a function that returns the application object to run.

Resource Files:

The application resource defines a significant part of how the application will appear and function. The resource file is a text file whose name ends in .rss, and is compiled into a binary form by the SDK's resource compiler. This compiled version of the resource file is loaded onto the phone along with the application executable and is accessed during application execution.

The RSS_SIGNATURE resource is used to validate the file and must appear, exactly as shown below, as the first resource in every appliation's resource file.

[cpp] RESOURCE RSS_SIGNATURE { }[/cpp]

TBUF Resource defines the default document name. As a document is not used in the application, it is left blank.
The EIK_APP_INFO Resource defines a Control Button Array(cba) resource that defines the function of the left and right softkeys. We define the cba as "R_AVKON_SOFTKEYS_EXIT". This defines the right softkey as exit and generates the command "EAknSoftkeyBack" when activated which is in turn handled by the Application UI class.

Application Registration Resource File:

GUI applications are EXE files in the sysbin directory and in order for the device to recognize an executable as a GUI application, and to display them on the desktop for user selection, the application must be registered via a special resource file known as a
registration resource file. The source file names are typically <application name> reg.rss. Walkthrough the reg_rss file for more details.

Project Build Files:

These are the files that define how to build a project:

FirstApp.mmp: It defines what source and resource files should be compiled and what libraries should be used for linking. The locations to search for project-defined and system include files, and source files and locations, are also defined

bld.inf: The file bld.inf points the build tools to the correct project definition
(MMP) file.

Download Source code for this tutorial here. Go to tutorial 2 now.

Building the Yellow Pages Application

Sun, 05 Jun 2011 20:25:06 GMT

Read the following chapters of the book, Developing Software for Symbian OS by Steve Babin. :

Chapter 1: Smartphones and Symbian OS
Chapter 2: Symbian OS Quick Start
Chapter 4: Symbian OS Programming Basics
Chapter 6: Strings, Buffers, and Data Collection (Important)
Chapter 12: GUI Application Programming

This would require about 8-10 days.

In addition to this, we also referred to Nokia Forums, NewLC and the book, "Quick Recipes on Symbian OS Mastering C++ Smartphone Development" by Michael Aubert. (Section 4.1 : File Handling)

After reading the book we started off with a top-down design of the App (Use case analysis, class design, realization etc).

Application Use Case Analysis

User selects City from a list.
User selects Category from a list.
System presents the list of Companies and their phone numbers to the user.

Application Architecture

A Symbian GUI App contains 4 classes:

Application View: The root GUI control, this class implements the main window and acts as a container for the other application controls.
Application UI: This class instantiates the application view and handles the commands sent from the application’s GUI controls.
Application document: This class handles the non-GUI data aspects of the application – the application data. It also instantiates the application’s UI class.
Application: The main application class starts the application by instantiating and starting the document class. It also sets the application’s UID.

The application resource defines a significant part of how the application will appear and function. The resource file is a text file whose name ends in .rss, and is compiled into a binary form by the SDK’s resource compiler. This compiled version of the resource file is loaded onto the phone along with the application executable and is accessed during application execution.

For making the GUI components, we must first define the GUI components in the resource file. For our Yellow Pages, we first define a MenuBar. We defined a menubar with just one item, "Select City". It should then display a dialog with a form to select the city-

Selecting a City and saving then creates another form which provides options for selecting the category-

After that, the data is read from a file and the names of companies and their phone numbers are dispayed in a MessageQueryDialog.

The UI class handles the commands from these GUI resources, whereas the CYPagesForm & CYPagesForm1 handle the dialog forms defined in the resource as follows. CYPagesForm & CYpagesForm1 have methods like DynInitMenuPane which initializes the resource defined in the resource files using the resource id, and other methods to handle the forms.

Google Summer of Code with Fluid Infusion

Wed, 01 Jun 2011 18:26:38 GMT

People who are waiting for my Symbian tutorials would have to wait. In the past couple of weeks, I haven’t had the time to write those tutorials as I have been working on a Google Summer of Code Project for the Inclusive Design Institute. For those who are unfamiliar with Google Summer of Code, it is a program that offers student developers the opportunity to write code for various open source projects. The project that I am working on is Image Editor which will be developed as a component for the fluid infusion framework. Fluid Infusion is a Javascript library that provides several components for building awesome interfaces for the web. I will be talking about a lot of things related to GSoC in this post. Lets get started.

Google announces GSoC every year sometime in January. It then accepts applications from several organizations who wish to work under the program and hire student developers for the summer. Few organizations get selected which then propose several projects on which the students would work on during the summer. This is when the students start their journey. I looked at several organizations and listed a few whose projects matched with my field of interest. The next step is to learn more about these organizations and make sure that you would like to work with them in the summer. It is very important to identify the correct organization and the correct project that you want to be working on. We can get to know more about the organizations by hanging out in their IRC channels or joining their mailing lists.

After this, I started writing proposals for the projects which I was interested to work on. This is probably the most important step of the application process. This is where the organizations can know about whether you have understood about the project and how much work do you propose to do. At this step, one could get feedback about the proposal from the mentor organization as to what they expect from the proposal. There is no such thing like a set of guidelines for writing a proposal because different organizations look for different things in the proposal. But, in general, a proposal should be realistic and to the point with at least a basic timeline of the work that you expect to complete every 15 days. One can submit upto 20 proposals to Google. Again this is an upper limit and is not at all recommended. Proposals should be submitted only to organizations which really interest you. Also one should focus on the quality of proposal rather than the quantity.

After this phase, one can either choose to relax till the declaration of results or study about the projects that he submitted proposals for. A few lucky ones make it to GSoC and it is an opportunity to get to know more about how large open source organizations work as well as contribute back to the open source community. This was just the beginning. The real work starts after being selected for GSoC. I guess, this is a long enough list of suggestions for the GSoC Applicants. I will now talk about my GSoC proposal and the wonderful organization that I have started working with.

You can find my proposal on Image Editor here. This is the final version of the proposal which I came up with after a few discussion with the mentor for Image Editor, Michelle D'Souza. I was very excited after I came to know that I had been accepted for Goolge Summer of Code. Frankly, it came to me as a shock. Thanks again to Fluid for giving me this wonderful opportunity.

During the community bonding period, I started looking up documentation on Fluid Infusion and remained in the IRC channel. This was the first time that I was working on a big open source project involving several developers. But, the Fluid community is really wonderful everyone is more than willing to help. With the help and support from my mentor as well as the developers, I was able to pickup the framework quickly and started developing simple components. I am working using github and all the code that I develop is available here. I am doing the development using JavaScript, HTML5, PHP and am working on Aptana Studio. I will be talking more about developing components for fluid and how to use fluid to develop good interfaces in my next posts.

In the meantime, I also received the Welcome Package for Google Summer of Code students. I was very excited to receive the package as it was from Google. Well, I guess that’s enough for today. I should probably get back to coding :). Have a look at my welcome package:

Step by step tutorial for your first Symbian Application

Tue, 24 May 2011 12:30:43 GMT

Welcome. This post will guide you about C++ programming for the Series 60 v5 platform, based on the Symbian Operating System. I wrote these tutorial along with my colleague Sapan, when we were doing summer internship at IIIT- Hyderabad. We built a simple application named "Yellow Page Application" under guidance of Dr. Rohit Agarwal. The term Yellow Pages refers to a telephone directory of businesses, categorized according to the product or service provided. This application does the work of displaying this information to the user on the go using just his mobile phone. In this way, he can get the information about the business he is looking for in a particular city.This application lists major businesses (Banks, Clinics and Doctors etc.) in 4 major cities of India.

We have divided this tutorial in 5 parts which will guide you How to build a basic Symbian application from scratch. Before going further I suggest you to get familiarize yourself with the application first and read this.

Tutorial Start Page : This part will describe about prerequisite setup needed for building any Symbian Application.

Tutorial 1: This part will tell you about 4 necessary classes that you must define in order to build Symbian Application.

Tutorial 2: This part will show you how to build a dialog box in Symbian Operating System.

Tutorial 3: This part creates forms and provides support for your application.

Tutorial 4: This part focuses on all the code that is required to build this application.

In each part I have provided a link to download source code also.

This tutorials took a long time so if you find any error in this tutorial please feel free to give your suggestions.

Tutorial Start Page

Tue, 24 May 2011 12:29:14 GMT

Symbian is a proprietary operating system designed for mobile devices, with associated libraries, user interface, frameworks and reference implementations of common tools, developed by Symbian Ltd. It is a descendant of Psion's EPOC.

A little history -- On 24 June 1998, Symbian Ltd. was formed as a partnership between Ericsson, Nokia, Motorola, and Psion, to exploit the convergence between PDAs and mobile phones. Symbian was previously owned by Nokia (56.3%), Ericsson (15.6%), Sony Ericsson (13.1%), Panasonic (10.5%), and Samsung (4.5%). Ten years later to the day, on 24 June 2008, Nokia announced that they intended to acquire all shares that they did not already own. Subsequently, Nokia opened up the development of Symbian to the broad industry by creating the Symbian Foundation.

In this tutorial,we describe how to get started with Symbian Programming by taking you through the installation steps, and then taking through the design of a sample application. This tutorial assumes that you have worked with C++ before and have a brief knowledge about classes, inheritence etc. If you are facing problems with these topics, please dont start this tutorial before making those concepts clear. This tutorial requires no previous knowledge about Symbian and will be taking you forward from the beginning. The first section in this tutorial describes the installation of the development environment for Symbian Programming and the second part starts from a top down class design for a sample application.

Getting Started: Hardware and SDK Installation

Part 1: The Hardware

The first thing you need is a PC/Monitor/mouse and Keyboard. Almost any machine on market today can be used for development, however some would save you a bit of time:

Don't pay too much attention to the CPU (just avoid if you can the entry level Celeron and Sempron) but take as much RAM memory as you can. 1GB seems a minimal requirement if you consider doing serious development, a big fast hardisk (Speed is important to reduce compilation times and emulator start time, space is a requirement to handle all your SDKs. As an example, you will need 1GB for the S60 3rd FP1 SDK, 500MB for Carbide.c++, etc...). also avoid Windows Vista for now, the ARM toolchain is not yet compatible with it. If everything is ok on the hardware/operating side, you are almost ready to start.

one last thing to check: make sure you have enough space on your C: drive. This is not an absolute requirement, but if you don't want to bother mounting virtual drive and editing configuration files again and again, you will need to install everything, the IDE, the SDK(s), your project(s) on C: so a minimum of 2GB free space is required before starting installation (and this is a minimum!).

Part 2: The software development toolchain

Now let's go for a bit of downloading. So download and install: ActivePerl v5.6.1. The latest 5.8 nor 5.10 versions won't probably work.

The Carbide.c++ IDE. : Carbide.C++ is a software development tool for C++ development on Symbian OS. It is used to develop phones that use the OS, as well as applications that run on that phone. It is based on Eclipse IDE Platform enhanced with extra plugins to support Symbian OS development. The Express version (which is free) is enough if you are a student or just intend to learn Symbian OS development. The Developer version (or better) is required if you plan to do some commercial development. To avoid any problem, be sure to install everything on C: in their default location.

The Software Development Toolchain: The SDKs

The S60 SDK : The S60 Platform (formerly Series 60 User Interface) is a software platform for mobile phones that runs on Symbian OS. S60 is currently amongst the leading smartphone platforms in the world. It is owned by Nokia. S60 is licensed by Nokia to other manufacturers including Lenovo, LG Electronics, Panasonic and Samsung.

Download the S60 SDK. Install the SDK in its default directory and on the same drive as Carbide.c++ At the end of the installation, you will be prompted to install the CSL Arm Toolchain if it is not already installed on your machine. Make sure you install it or you won't be able to compile for the phone target.

Reboot your machine

The Software Development Toolchain: Checking the installation

At the command prompt type 'epoc'. The S60 emulator window should come up.

The Software Development Toolchain: Reference Book :

Developing Software for Symbian OS by Steve Babin.

The Software Development Toolchain: Getting Help

You can refer to following links for help in Symbian Programming

For more help, you can look up the examples provided with Carbide.C++ and the SDK. Here are the instructions to running the Hello World example in Carbide.C++:

Run Carbide.C++
Goto File -> Import

Select Symbian OS -> Symbian OS Build.inf File and click on Next.Then browse to the directory where you installed Symbian SDK, and look for Examples in the subdirectories. The default location should be : C:Symbian9.2S60_3rd_FP1_3Examples. Once there, browse through Basics -> HelloWorld. Select the bld.inf file and proceed further.

Proceed through the next screens checking all the check boxes and then Run the Project by Pressing Ctrl+F11 or by selecting Run -> Run.

Image Inpainting

Sun, 09 Jan 2011 15:43:48 GMT

Image Inpainting is the art of filling in lost portions of an image using background details in a visually plausible way. I came across this topic while searching for projects in the field of image processing for sixth semester. The moment I saw a few sample pictures, I decided that I was gonna work on it.

As is quite clear from these images, a good and fast algorithm for Image Inpainting can be really useful. I worked with Sapan to develop the software. We started with a traditional algorithm but finally ended up making several changes to it in order to improve the results.

Here’s a brief decription of the inpainting algorithm.

Divide the marked region into patches.
Assign priorities to the patches (Using Data and Confidence Terms)
Choose patch with maximum priority and find a patch from background region that is most similar to the chosen patch
Replace the patch with the patch from background and update data and confidence values.

This is a very brief overview of the inpainting algorithm. For more details, you may feel free to contact me or go through the research paper that we authored.

For more information about the implementation, visit our project page on Sourceforge. I hope that you would enjoy using it. It is still a very early release and we hope to improve on it soon. Looking forward to your participation on the project as well.

See the software in action in this video.

https://www.youtube.com/watch?v=DHF6xX83uhM

Stay tuned for my next post. Meanwhile, tell me your suggestions/comments about my work.

Gaming Zone!

Wed, 01 Dec 2010 00:04:02 GMT

Ever since my childhood, I have been very interested in computers, especially PC gaming. It all started with the small DOS based games like Prince of Persia and Dave. I still remember beating all the levels in dave again and again. I wish I could get my hands on those games again. That's how my love for computer gaming began. Over the years, I have played several games. I have seen these games develop and the graphics, visual effects and artificial intelligence has improved dramatically.

But my real gaming started only after getting admission into IIIT. First time when I saw my seniors play counter strike, I was amazed. It was so great watching all of them play together, shouting at each other, killing each other (in game of course :P). I was dying to play it. I started playing with a long name which eventually shortened down to (J5. I was enjoying every moment of it. The team work, the strategy and the thrill. We used to play whole day and night.

Meanwhile, my interest into Age of Empires 2: The Conquerers also developed into addiction. It was again thanks to the seniors, specially, Red Bull, Sa61e and SEXYSUNNY. I enjoyed watching their games and hoped that I would play like them one day. :) I was among the first few who started playing AOE in my batch, the others being Sapan, Varun and Arun. I enjoyed playing AOE and CS a lot. We played a lot with each other and with the seniors and super seniors. I still remember the game that I and Sapan played against LunePredator and Vivek Sir. We had no idea how good they were at age and thus started playing a 2 on 2 match with them. And as it always happens in matches between seniors and juniors, we were brutally destroyed :(. That's how our journey began with AOE and we gradually improved and started giving good fight to our seniors :D.

Enough about AOE. I also enjoyed playing FIFA. My interest for FIFA had begun long before I had joined the college. But after joining IIITA, it grew much larger. I enjoyed playing it on LAN with my colleagues. I participated in the tournament that seniors organized as well as our batch tournament and enjoyed every single match. Gaming has always been my passion and will remain for a long time. PC gaming is a great experience and I believe that it is something that everybody should try to do at least once. :D

Joining Sun Family

Mon, 29 Nov 2010 18:19:41 GMT

This story begins with a mail from our Dean Student Affairs, Prof. G.C. Nandi. I was told to send my resume to him. At that time, I was unaware about what it was for. Later, I came to know that it was for the post of Sun Campus Ambassador for my campus. I was very excited about it. But at the same time, I was worried. It was the first interview that I was going to face. They had scheduled a telephonic interview for me with a Sun Engineer. I was in my third semester then and didn't know much about interviews. I had no idea about what it was going to be like.

So, in complete ignorance, I waited for the day of interview. She started the interview with some general questions and then slowly came towards technical questions. It was a nice experience which lasted for about 30 minutes. After a few days, I received a call telling that I had been selected as the Sun Campus Ambassador. I was very excited. I had never dreamed that I would be promoting Sun Microsystems in my campus. With this began my wonderful journey as a Sun Campus Ambassador. :)

After a few days, I received a letter from Sun Microsystems about my responsibilities and my salary ;). This was how I became a part of the Sun Microsystems family. I began by organizing a few presentations about Sun products. I was amazed at the wonderful responses of my colleagues. I also arranged a special screening of Glassfish Webinar organized by Sun. Motivated with the success, I along with my roommate Sapan, organized a Free and Open Source Software (FOSS) Quiz. It attracted a lot of students. Our complete B2k7 batch and most of my seniors and juniors participated in the quiz. All the prizes, including 6 pen drives for the top 3 teams and various other goodies were sponsored by Sun Microsystems.

I enjoyed every moment as a Sun Campus Ambassador. After the merger of Sun and Oracle, I acquired the role of OSUM Affiliate Leader for my campus as a part of which I promoted various Sun and other Free and Open Source Softwares. We also arranged another FOSS quiz which was also very successful.

Blog by Pulkit Goyal

An Introduction to Exceptions in Elixir

Raising Exceptions in Elixir

Handling Elixir Exceptions

Re-raising Exceptions

Rescue And Catch in Elixir

After and Else Blocks in Elixir

Exceptions and Processes

Monitoring Exceptions

Wrapping Up

An Introduction to Mocking Tools for Elixir

Test Without Mocking Tools in Elixir

Test Stubs

Roll Your Own Server

Elixir Mocking Tools

Mock

Mox

Mock the External Client

Test the External Client

Mimic

Use Mocking Right for Your Elixir App

Wrap Up

Database Performance Optimization and Scaling in Rails

Disclaimer: Scaling Isn’t Always Needed

A Separate Database for Your Rails Application

Using Read-Only Replicas for Rails

Setting up a Read-Only Replica

Gotchas

Database Sharding in Your Rails Application

Setting up Horizontal Sharding

Wrap Up

State Machines in Ruby: An Introduction

State Machines in Development

Our First State Machine in Ruby

Using the State Machines Gem in Ruby

Execute Side-Effects with the State Machines Gem in Ruby

Use the State Machines Gem with ActiveRecord in Ruby on Rails

Wrap Up: Build State Machines in Ruby

Using Profiling in Elixir to Improve Performance

Do I Need Profiling for My Elixir App?

Profiling with cprof in Elixir

eprof Profiling in Elixir

Using fprof in Elixir

Profile and Collect Metrics from Your Elixir App in Production

Phoenix Live Dashboard

Recon

AppSignal

Wrap-up: Measure Your Elixir App’s Performance with Profiling

An Introduction to Pattern Matching in Ruby

Ruby Pattern Matching with case/in

Pattern Matching Arrays in Ruby

Pattern Matching Objects in Ruby

Variable Binding and Pinning in Ruby

Matching Ruby’s Custom Classes

Using Guards for Complex Patterns

Pattern Matching with =>/in Without case

Pattern Matching in Ruby: Watch This Space

What's new in Rails 7

Node and Webpack Not Required

Turbolinks and UJS Replaced by Turbo and Stimulus

Encryption at Database Layer

Asynchronous Querying

Zeitwerk Mode for Rails 7

Other Rails 7 Updates

Retry Jobs Unlimited Times

Named Variants

Hash to HTML Attributes

Ruby debug

Assert a Single Record with sole

Check Presence / Absence of an Association

Stream Generated Files from Controller Actions

Upgrading to Rails 7

Wrap up

Testing WebSocket Clients in Elixir with a Mock Server

Introduction

WebSocket Client

Testing External Services

Mock Server

Mock HTTP Server

Mock Socket

Profiling with `cprof` in Elixir

`eprof` Profiling in Elixir

Using `fprof` in Elixir

Ruby Pattern Matching with `case`/`in`

Pattern Matching with `=>`/`in` Without `case`

Ruby `debug`

Assert a Single Record with `sole`

`useCounter.js`

`useCounter.test.js`

`useCounter.js`