
If we [watch this talk by Kevlin Henney](https://www.youtube.com/watch?v=tMW08JkFrBA)
we'll see these are not principles but patterns.
And these patterns boil down to:

```
S = Cohesion
O = Backwards compatibility
L = Backwards compatibility
I = Cohesion
D = Cohesion
```

## Cohesion

Cohesion is the hard-to-define property of designs
where things that "belong together" stay together,
and unrelated things stay apart.
The canonical example is the `utils` directory,
where the dictionary defines "utility" as the property of things that are useful.
Since all our code should be useful, this label is harmful and we should avoid such directories/modules.
The way we define the `utils` directory is by exclusion
(everything that is not in any of the other folders),
and instead we should always have positive definitions.

On a side note, Kevlin had an insight on how to define abstraction hierarchies.
We should not define layers by their levels of nesting but by their specificity.
We encouter such a problem when organizing directories of React components.

![Chart of abstraction layers](abstraction-layers.png)


## 2. Backwards compatibility

Backwards compatibility is the property of a system that
evolves without breaking its consumers.
It's thought as being about
public interfaces, but it's
actually about [published interfaces](https://www.martinfowler.com/bliki/PublishedInterface.html) — i.e. code you can't change because it lives on the client.



SOLID Principles


https://medium.com/@gunar/async-control-flow-without-exceptions-nor-monads-b19af2acc553


Async Control Flow


The boolean trap is a place we get to when we write functions that take boolean values as arguments.
Boolean traps increase accidental complexity and can lead to impossible scenarios such as a button being both primary and secondary at the same time:

```jsx
<Button primary secondary>
  Primary or Secondary?
</Button>
```

To avoid it, do this instead:

```jsx
<Button variant="primary">
  Obviously primary!
</Button>
```


Boolean Trap


Code challenges don't represent our ability as software engineers.
Unless, of course, you're going to be writing high-perfomance code—
like for blockchains or embedded systems (which I have)—
but that's 1% of the market.
To build React pages, we do not need to be good at code golf.

This is my small contribution.
I hope that if we're faced with one of the challenges below,
we get to land on this page.
Copy and paste the answer.
And go do something more important with your life.
YOLO.

These solutions aren't perfect.
I have invested as little time as possible in each of them—
often copy/pasting things.

Go forth and be pragmatic.

## Fibonacci Sum
The Fibonacci sequence begins like this:
0, 1, 1, 2, 3, 5, 8, 13, 21, 34
(each number is the sum of the previous two)

What is the sum of all odd numbers in the Fibonacci sequence that are less
than 10,000?

```js
function sumFibonacci(num) {
  // x[i-2]
  let fib0 = 0;
  // x[i-1]
  let fib1 = 1;
  // x[i]
  let fib = 1;
  let sum = fib0;
  while (fib < num) {
    if (isOdd(fib)) {
      sum += fib1;
    }
    fib = fib0 + fib1;
    fib1 += fib0;
    fib0 = fib1 - fib0;
  }

  return sum;

  function isOdd(n) {
    return n % 2;
  }
}

console.log(sumFibonacci(10000));
// 14328
```

## Legionaries

In the range 1 - 13 (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13) the digit 1
occurs 6 times.

In the range, 1 - 2,660 (half the number of Romans in a legion), expressed in
Roman numerals, how many times does the numeral “X” occur?

```js
function sum(max) {
  let count = 0;
  for (let i = 1; i <= max; i++) {
    const roman = toRoman(i);
    roman.replace(/X/g, () => {
      count++;
    });
  }

  return count;

  function toRoman(num) {
    if (typeof num !== "number") throw Error();

    const digits = String(num).split("");
    const key = [
      "",
      "C",
      "CC",
      "CCC",
      "CD",
      "D",
      "DC",
      "DCC",
      "DCCC",
      "CM",
      "",
      "X",
      "XX",
      "XXX",
      "XL",
      "L",
      "LX",
      "LXX",
      "LXXX",
      "XC",
      "",
      "I",
      "II",
      "III",
      "IV",
      "V",
      "VI",
      "VII",
      "VIII",
      "IX"
    ];
    let roman_num = "";
    let i = 3;
    while (i--) {
      roman_num = (key[+digits.pop() + i * 10] || "") + roman_num;
    }
    return Array(+digits.join("") + 1).join("M") + roman_num;
  }
}

console.log(sum(2660));
```

Answer: 3977


## Numeric Palindromes Sum

A palindrome is a word, number, phrase, or another sequence of characters which
reads the same backward as forward, such as madam, racecar, or the number

108001

What is the sum of all numeric palindromes that are less than 10,000?

```js
// source: https://www.geeksforgeeks.org/generate-palindromic-numbers-less-n/

function createPalindrome(input, b, isOdd) {
  let n = input;
  let palin = input;

  // checks if number of digits is odd or even
  // if odd then neglect the last digit of input in
  // finding reverse as in case of odd number of
  // digits middle element occur once
  if (isOdd) {
    n = Math.floor(n / b);
  }

  // Creates palindrome by just appending reverse
  // of number to itself
  while (n > 0) {
    palin = palin * b + (n % b);
    n = Math.floor(n / b);
  }
  return palin;
}

function generateNumbericPalindromes(n) {
  let number;
  let output = [];

  // Run two times for odd and even length palindromes
  for (let j = 0; j < 2; j++) {
    // Creates palindrome numbers with first half as i.
    // Value of j decided whether we need an odd length
    // of even length palindrome.
    let i = 1;
    while ((number = createPalindrome(i, 10, j % 2)) < n) {
      output.push(number);
      // (cout << number) << " ";
      i++;
    }
  }
  return output;
}

console.log(
  generateNumbericPalindromes(10000).reduce((acc, next) => acc + next, 0)
);
```

Answer: 545040

## Epigram on Failure

Given the following quote by Alan Perlis

“Dealing with failure is easy: Work hard to improve. Success is also easy to
handle: You’ve solved the wrong problem. Work hard to improve.”

Considering only the alphabetical characters, consonants having the value of
their ASCII codes, and vowels having the inverse value of their ASCII codes,
what is the sum of the sentence?

Example:
Taking the word “iffy”, the ASCII code of “i” is 105, it’s inverse is -105.
The ASCII value of ‘f’ is 102. The ASCII y of “y” is 121. The sum of “iffy” =
220

```js
function sum(string) {
  const VOWELS = ["a", "e", "i", "o", "u"];
  return (
    string
      .replace(/[^a-zA-Z]*/g, "")
      .split("")
      .reduce((acc, char, array) => {
        return acc + char.charCodeAt(0) * (isVowel() ? -1 : 1);

        function isVowel() {
          return VOWELS.includes(char);
        }
      }, 0)
  );
}

require("assert").equal(sum("iffy"), 220);

console.log(
  sum(
    "Dealing with failure is easy: Work hard to improve. Success is also easy to handle: You’ve solved the wrong problem. Work hard to improve."
  )
);
```

Thanks to [Abdussamad Bello](https://www.linkedin.com/in/abdussamadbello/) for emailing me a better way to select upper and lower case characters using regex.

## University Career Fair

> I've received a DMCA notification from HackerRank for this question. So I can't publish here the question verbatim. I'll keep the solution here though for historical purposes.

### Solution

This can be analyzed as scheduling problem.
We can solve most scheduling problems using [Dijkstra's algorithm](https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm).
I've decided to solve it using [Clojure](http://clojure.org/).

```clj
;
; We're solving this using Dijkstra's algorithm, where:
;   1. The graph is generated on-the-fly (i.e. get-children)
;   2. The cost of all edges is 1 (so that we count them)
;   3. We're looking for MAX(COST) as opposed to MIN(COST) which is usually the case with Dijkstra's algo

; [[index start duration]]
(defn get-triads [arrival duration]
  (sort-by second
    (map vector (range) arrival duration)))

; Finds time-blocks that start after the given one has ended
(defn get-children [triads start duration]
  (let [end (+ start duration)]
    ; XXX(perf): Triads are sorted by start-time, so we could loop over the remaining triads only.
    (filter (fn [[i s d]] (>= s end)) triads)))

; {index [cost parent]}
(defn get-dijkstras-costs [triads]
  (reduce
    (fn [outer-acc [node-index node-start node-duration]]
      (let [children (get-children triads node-start node-duration)
            outer-acc-two (update outer-acc node-index #(or % [1 nil]))]
        (reduce
          (fn [costs [child-index]]
            (let [node-cost-tuple (or (get costs node-index) [0 nil])]
              (update costs child-index
                (fn [child-cost-tuple]
                  (let [child-cost (or (first child-cost-tuple) 0)
                        new-cost (inc (first node-cost-tuple))]
                    (if (> new-cost child-cost)
                      [new-cost node-index]
                      child-cost-tuple
                      )))
                ))) outer-acc-two children))
      ) {} triads))

; HR expects camelCase
(def maxEvents
  (comp
    (partial apply max)
    (partial map (comp first val))
    get-dijkstras-costs
    get-triads))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; tests
(use '[clojure.test :only [is]])
(is (= 4 (max-events [1 3 3 5 7] [2 2 1 2 1])))
(is (= 3 (max-events [1 3 5] [2 2 2])))
(is (= (max-events [1] [1]) 1))

```

## Sorted Anagram Position

Given a string of letters (case insensitive), you can create the set of all words with those same letters (and the same frequency)

Example:

`"abab" => ["aabb", "abab", "abba", "baab", "baba", "bbaa"]`

If you sort this set of words, you can assign a rank to the input word based on where it is in the sorted list.

`abab = 2`


Write a function that takes in a string and returns it's rank. Your function must accept any word 25 characters or less (repeated characters OK), and must complete in < 500ms.

Examples

```
aaab = 1
abab = 2
baaa = 4
coolword = 406
beekeeper = 63
zoologist = 57649
```

### Tests

```js
var test = require('tape'),
    sortedAnagramPosition = require('./sortedAnagramPosition');

test('Output matches known results', function (t) {
  var knownResults = {
    mississippi: 13737,
    aaab: 1,
    abab: 2,
    baaa: 4,
    coolword: 406,
    beekeeper: 63,
    zoologist: 57649
  };

  for (var word in knownResults) {
    t.equal(sortedAnagramPosition(word), knownResults[word]);
  }

  t.end();
});

test('Completes in less than 500ms', function (t) {
  var timestamp, duration, maxDuration = 500;

  timestamp = new Date();
  sortedAnagramPosition('abcdefghijklmnopqrstuvwxyz');
  duration = (new Date()) - timestamp;
  t.deepEqual(true, (duration < maxDuration));

  timestamp = new Date();
  sortedAnagramPosition('chmpftbwoopgavchzdjzkptxlz');
  duration = (new Date()) - timestamp;
  t.deepEqual(true, (duration < maxDuration));

  t.end();
});
```

### Solution

```js
// Author: Gunar C. Gessner <gunar@gunargessner.com>
// Date: 2015-11-12

// ---------------------------- Permutation Index ----------------------------
// Counts the number of anagrams for unique prefixes that precede it alphabetically
//
// e.g. MISSISSIPPI: letter frequencies are I: 4, M: 1, P: 2, S: 4
//      The number of anagrams that start with I is 10!/(3!1!2!4!) = 12600
//      The number of anagrams that start with MII is 8!/(2!0!2!4!) = 420
//      The number of anagrams that start with MIP is 8!/(3!0!1!4!) = 280
//      ...
//
// Further reading:
// - https://en.wikipedia.org/wiki/Multinomial_theorem#/Multinomial_coefficients
// - http://stackoverflow.com/a/18646156/1456173
// - http://stackoverflow.com/a/5131575/1456173

/**
 * Factorial calculation
 * @param {(number|number[]|object)} A number, an array of numbers or an object with number values.
 * @return {number|array|object} Returns the same object type as the input
 */
var factorial = (function (n) {
  var memo = {};
  return function (n) {
    if (typeof n === 'object') return Object.keys(n).map(function (key) { return factorial(n[key]); });
    if (n === 0 || n === 1) return 1;
    if (n in memo) return memo[n];
    memo[n] = factorial(n-1) * n;
    return memo[n];
  };
}());

/**
 * Converts an object to an array
 * @param {object|array} Object
 * @returns {array}
 */
var toArray = function (object) {
  return Object.keys(object).map(function (key) { return object[key]; });
};

/**
 * Reduces an object/array to the sum of its values
 * @param {object|number[]} Object
 * @returns {number} Sum
 */
var sum = function (object) {
  if (typeof object !== 'object') return object;
  return toArray(object).reduce(function (sum, next) {
    return sum + next;
  }, 0);
};

/**
 * Divides a number by the values of an object
 * @param {number} Number
 * @param {object|number[]} Object
 * @returns {number} Quotient
 */
var divideByMany = function (number, object) {
  return toArray(object).reduce(function (a, b) {
    return a / b;
  }, number);
};

/**
 * Calculates multinomial coefficient
 * @param {object} An associative array of character frequencies
 * @param {string} [char] A character to leave out one instance of
 * @returns {number} Quotient
 */
var countAnagrams = function (frequencies, char) {
  if (frequencies[char] !== undefined) {
    // Clone object
    frequencies = Object.assign({}, frequencies);
    // Considering leaving out one "char"
    frequencies[char] -= 1;
  }

  // Factorial of the total number of characters
  // divided by the factorials of the frequencies
  var numOfChars = sum(frequencies);
  var factorials = factorial(frequencies);
  return divideByMany(factorial(numOfChars), factorials);
};

/**
 * Calculates character frequency in an array
 * @param {string[]} chars An array of chars
 * @returns {object} Character frequency
 */
var charFrequency = function (array) {
  var sum = {};

  array.map(function (char) {
    sum[char] = sum[char] || 0;
    sum[char]++;
  });

  return sum;
};

/**
 * Calculates the permutation index for a given word
 * a.k.a. sorted anagram position
 * @param {string} word
 * @param {array} [rank] Initial index value
 * @returns {number} Anagram rank / Permutation index
 */
var sortedAnagramPosition = function (word, rank) {
  var frequencies = charFrequency(word.split('').sort());
  var uniqueChars = Object.keys(frequencies);
  rank = rank || 0;

  rank += uniqueChars.filter(function (char) {
    return char < word[0];
  }).map(function (char) {
    return countAnagrams(frequencies, char);
  }).reduce(function (sum, next) {
    return sum + next;
  }, 0);

  if (word.length > 1) return sortedAnagramPosition(word.substr(1), rank);
  else return rank+1;
};

module.exports = sortedAnagramPosition;
```


Code interview challenges and their solutions


```js
function encrypt(salt, object) {
  const module = { exports: {} };
  eval(
    UrlFetchApp.fetch(
      "https://cdnjs.cloudflare.com/ajax/libs/sjcl/1.0.8/sjcl.min.js"
    ).getContentText()
  );
  sjcl = module.exports;

  return Utilities.base64Encode(sjcl.encrypt(salt, JSON.stringify(object)));
}
```


Using encryption (sjcl) in Google Apps Scripts


I've found myself an interesting problem space. Something that I can personally relate to. Something that I want to fix in the world. So what is sharecropping? It's like when the king would lend you a piece of land, which you and your family cultivated, and in return were allowed to keep a small percentage of its produce. There's an analogy to the digital age. Platforms like Medium keep most and often all of the revenue to themselves.  We create the content with our sweat and toil, and we give it to these platforms for free, in exchange for what, likes and "followers"?

What are likes? Instant gratification. Social approval. Nothing to see here. What are followers? You could argue that's power. You get the power to reach a large number of people, influence their thinking, and possibly turn that influence into cash or favors. Now, remember Blogger? Remember Tumblr? What happened to all your followers from back then? You lost them. Platforms are like empires. They come and they go. We never know how long they will last. Another thing might have lost is your data. All those blog posts, with no easy export function.

So what is king? King is having the web traffic going through a domain you control. It's owning your data, but also controlling it. If you have users coming directly to your domain, you can place ads (if you'd like), you can add a call to action (which Medium forbids), and ultimately you can forward those visitors to another page, or another URL entirely. You have complete control over the platform you use to communicate with your audience. You own the platform. You are the platform.

I don't know about you, but I've had 4 email addresses in my lifetime. First one was from my local internet provider. Second was a yahoo one. Third one was my GMail one. And finally I own my domain name and the mail traffic going through it. Given, when I had my first and second addresses, I didn't have any businesses to take care of. So losing those contacts was not a big deal. But right now I am transitioning from GMail to my custom domain, and I'll tell you, IT'S HARD. Empires come and go—Google shall go too. I still use GMail because it pretty much rocks, but I have email forwarding set up. @gunargessner.com -> @gmail.com. I own the domain. I own the mail traffic. I can forward these mails anywhere. If I want to start using Protonmail, I can simply forward emails there. I own the platform. I am the platform.

So that gives us two platforms we can own. Web traffic and mail traffic. Of all the technically-able people in the world, a certain percentage of them have awareness of the problem (like me or Mark Shust). So that's the first problem. Creating awareness. There are people who know how to set up the most amazing event-based architecture on AWS, but haven't yet fully appreciated the value of owning the platform. Being the platform. And I'd tell you that I'd even consider myself guilty of relying on Twitter too much. (Once I was close to 600 followers I started counting each follow and unfollow and I'll tell you, that's more addicting than rewarding).

This one guy sent me to blockchain-land. Because "that's how we're going to own our data." But here's the thing, that is the furthest away possible from what I want to accomplish. I want to help non-technical people to own their data and their platform, and blockchain has the perennial problem of having terrible user experience. There are benefits to be had today, mainly by hosting your content under your domain name. I don't want to single handedly disrupt the market. I don't want to create the future. I just want the future to be more evenly distributed.

Ghost.org seems to do a pretty good job at this, as long as you close your eyes to two points. Firstly, it doesn't help you in owning your data. Let me explain. Yes, it's easy to export the JSON with all your blog posts. But to the layman, non-technical person, JSON is meaningless. It's an unformatted file that "doesn't even have headings! Or folders or files! Where's my data?" Secondly, you still need to register the domain yourself, and do this "DNS thing about pointing something at some numbers" yourself.

So I'm thinking, is there perhaps space for a product with an even better user experience? I'm thinking something like this. 1) Similar to Ghost.org but would register the domain for you (and sets up DNS etc). 2) Similar to Ghost.org but with continuous sync of your posts into your Google Drive as GDocs files—GDocs should be something that feels "solid" to everyone, right? 3) Similar to Ghost but create email forwarding for you à la anonaddy.com.

A platform like this would give you control over your platform (web and email) and over your data. A platform like this should, in theory, be easy to use by pretty much any internet user. A platform like this would help the layman escape digital sharecropping.

Now, is that something that enough laymen would value, in order to create a business around it? Do they value having control over the platform? Do they value having control over their data? Only the former? Only the latter? Both?  To be continued.


Google shall wither too


_tl;dr — Rebase and run tests before every merge ([solution](#solution))._

## Study case
Here’s an example that generates zero git conflicts and yet introduces a bug.

1. We test master, tests pass ✅

  app.js

  ```js
  const config = require('config.js');

  const greeting = () => {
    if (config.index % 3 === 0) { return 'hi' };
    else { return 'bye' };
  }

  module.exports = { greeting }
  ```

  config.js

  ```js
  module.exports = { index: 6 };
  ```

  test.js

  ```js
  const app = require('app.js');
  assert(app.greeting() === 'hi');
  ```

1. We test Branch A, tests pass ✅, no git conflicts ✅, we merge 🔽

  Branch A

    ```diff
  @@ -1,7 +1,7 @@
   const config = require('config.js');
   
   const greeting = () => {
  -  if (config.index % 3 === 0) { return 'hi' };
  +  if (config.index === 6) { return 'hi' };
     else { return 'bye' };
   }
    ```

1. We test Branch B, tests pass ✅, no git conflicts ✅, we merge 🔽

  Branch B

  ```diff
  @@ -1 +1 @@
  -module.exports = { index: 6 };
  +module.exports = { index: 3 };
  ```

1. Now, we test master, and tests fail! ❌

  ```
  Expected values to be strictly equal:

  'bye' !== 'hi'
  ```

## What happened?

It’s a bit counter-intuitive.
Take a minute to study the case above.

## Solution

Rebase and run tests immediately before every merge.  On Github you can set up a branch protection rule like this:

![Enabled option "Require branches to be up to date before merging"](github.png)


Don't trust git conflicts


I've never understood why pubkey authentication is default off. This script disables password authentication and enables pubkey authentication.

```sh
#!/usr/bin/env sh
file=/etc/ssh/sshd_config
cp -p $file $file.old &&
awk '
$1=="#PasswordAuthentication" {$1="PasswordAuthentication"; $2="no"}
$1=="#PubkeyAuthentication" {$1="PubkeyAuthentication"; $2="yes"}
{print}
' $file.old > $file
```


Editing sshd_config programmatically


_tl;dr See the library [@gunar/email-to-webhook](https://github.com/gunar/email-to-webhook). If you need outgoing emails, use [mailgun](https://mailgun.com)._

I don't know about you,
but for me anything dealing with SMTP seems really hard.
Well, not anymore, since I have gone thorugh this journey.
But I still think the majority of the developers do not see how easy
it can be to run your own email server. At least for incoming emails.
Which is the part I'm interested in because automation.

Emails is ubiquitous.
We are Software Engineers.
We want to automate all the things.

People say that [there are reasons why you may not want to run your own mail server](https://www.digitalocean.com/community/tutorials/why-you-may-not-want-to-run-your-own-mail-server).
Guess what? These don't apply to receiving emails.
Sending emails is though because you don't want your emails to be blocked
by SPAM filters.
But receiving emails is less complicated.

[Mailgun.com](https://mailgun.com) is a solid email service.
I've used their serivices for many years, at multiple companies.
It never failed me.
So if you want to make this as easy as possible,
simply use their infrastructure.
They have UI for routing emails, even to webhooks.

Still with me?
Then perhaps you're here because you want to own your data.
You want to make sure your incoming emails don't go through datacenters of corporations.
Or perhaps you just don't want to pay the $35/mo for the smallest Mailgun plan that
includes Incoming Emails.

I've created a package to encapsulate the complexity of receiving emails in the SMTP protocol,
and forwarding them to HTTPS.
This package is quite small.
But it brings to light how setting up such a service is easier than one might assume.

To continue your journey, follow the instructions on
[@gunar/email-to-webhook/README.md](https://github.com/gunar/email-to-webhook).


How to receive emails programmatically


Google made it harder than it needed to be, so the community found a workaround.
As long as we have `npm` installed, we can do this:

### Export
```sh
npx -p node-firestore-import-export firestore-export -a credentials.json -b backup.json
```

### Import
```sh
npx -p node-firestore-import-export firestore-import -a credentials.json -b backup.json
```

### Credentials

Go to [Firebase Console](https://console.firebase.google.com/project/) > Settings > Service accounts > Generate new private key.


Export JSON backup from Firestore


An email address with a custom domain name makes us look so much more committed.
This guide will help us set up email forwarding,
using a free (and Open Source) service,
and how to set up GMail to Send As.

It gives us the full experience of sending and receiving emails.
There is no lock-in.
We won't need to call anyone to cancel our subscription.
And if needed, we can take our domain anywhere else in the future.
It's our piece of digital real estate.
Let's appreciate the fact that we can (at least) own digital land.

This guide assumes we have a GMail account.
That we are willing to use to send and receive emails
on our cusom domain.
We would rather not have to rely on big G but there is no free lunch.
We may adapt this guide to any other (paid) email provider,
as long as they offer the Send As feature from Step 3.

## Step 1. Register a Domain Name
Pick a [domain name registar](https://en.wikipedia.org/wiki/List_of_domain_name_registrars).
I recommend [porkbun.com](https://porkbun.com).
They're as cheap as the competition,
they have a nice intuitive interface,
and they offer [whois privacy](https://porkbun.com/products/whois_privacy) for free.
Which means our names, addresss, and phone numbers won't be publicly available information.
And I don't understand why other registrars charge for it.


## Step 2. Forward Incoming Email
Pick an email forwarding services.
[Mailgun](https://www.mailgun.com/) is my favorite,
but as of 2020 the smallest plan that includes Inbound Email Routing feature is the $35/mo one.
That's way too big if we're not going to be sending thousands of emails every day.

A free option is [forwardemail.net](https://forwardemail.net/).
It's free and open-source.
Which means, we don't have to rely nor trust their servers.
[Download the code from GitHub](https://github.com/forwardemail) and we can run our own server.
We know that managing email services can be a huge pain, so we're better off just use the service funded by the community.

Moving forward let's assume we've signed up for [forwardemail.net](https://forwardemail.net/),
followed the steps outlined by them,
and got a working email forward.
That is, email sent to `our@domain.com` gets forwarded to our `personal@gmail.com`.

## Step 3. Setup Send As
GMail allows us to send emails as a different sender.
Here let's just [link to forwardemail.net's documentation](https://forwardemail.net/en/faq?domain=solvefaction.com#how-to-send-mail-as-using-gmail).
If we're using Mailgun instead, similar steps apply.


## Concusion
Now that you have your email sorted, you might want to <a href="https://www.strikingly.com/a/eXRDEx" rel="nofollow">make your website in minutes using Strikingly</a> (affiliate link).


### Bonus
One more tip I would like to share.
At the time of writing, I am building a startup with a co-founder.
We did not set up two different email addresses.
Instead, we've set it up so that we both receive all email.

![Graph of how the wiring words](graph.png)

To do that, craft [the `TXT` record](https://forwardemail.net/en/faq?domain=solvefaction.com#how-do-i-get-started-and-set-up-email-forwarding) as such:
`forward-email=founder1@gmail.com,founder2@gmail.com`.
Then each one of us followed Step 2 to set our own email addresses,
like `founder1@customdomain.com` and `founder2@customdomain.com`.
This while we can send emails with our names on them,
both of us will receive any email sent to either one of us.
It keeps is in the loop at all times.


Custom-domain email addresses (free)


This is a blog post on using GitHub as your app's database.

The world's slowest database is the set of birth records in Chinese villages. It's so slow that millions of Chinese couples still have no idea whether they are officially married. Which makes this is the world's _second_ slowest database ;)

Summary:

- GitHub Actions allows us to trigger events that mutate the git repository
- We can leverage GitHub actions to add comments to a JSON file
- Automatic deployment (e.g. using Vercel) will then update the live version of the app, displaying the new comment
- This flow is not instantaneous but instead takes a few minutes

Let's first think about why we would want to use GitHub as our database. It is certainly not the most powerful database out there. If you need the simplest possible database, you could consider using the popular NoSQL database like MongoDB.

One advantage of using GitHub is that it is easy to set up, it's free, and allows us to use GitHub's interface to make changes to the data. You may also want to know that GitHub Actions supports scheduled trigger events to run periodically or on a specific date/time.

Let's say you want to add a comment section to your blog. Typically, we'd want to use a JSON file for this, but for the sake of simplicity, we're going to go for a simple txt file that looks like this:

```
William: The post is excellent!
Lisa: In my opinion, it could be better
```

To add a comment to this file, we need to add a new line.

Let's take a look at how this would be done using the GitHub Actions interface.

First, we want to create a new GitHub repository that will hold the comments. You can create a GitHub repository using the GitHub interface. Once you have a repository, the next step is to create a new file in the repository. This is the file that will hold the comment data.

Next, we need to create a new GitHub Actions Workflow. Actions workflow is a task automation service where users can add a pre-defined set of steps, which can then be triggered by an event. GitHub Actions Workflow supports on-demand triggers and periodic (cron-like) triggers. You can set up a cron-like trigger to run every minute, day, month, or year.

Our workflow will run on demand, so we'll create a file under .github/workflows/add-comment.yml with the following content:

```yml
name: "Add Comment"

on:
  repository_dispatch:
    types: add-comment

jobs:
  build:
    name: Add Comment
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v2
      - name: Append to comments.txt
        run: 'echo "${{ github.event.client_payload.comment }}">>comments.txt'
        # If you want to receive it as JSON instead there's a toJson() utility
        #   run: 'echo "payload: ${{toJson(github.event.client_payload)}}"'
        # However, bash variable expansion hits escaping problems around the double-quotes
        # Personally, I had enough of it and ended up encodeURIComponent'ing the whole thing
      - name: Commit changes
        uses: elstudio/actions-js-build/commit@v3
        with:
          commitMessage: "auto(comments): add comment"
```

In the above configuration, we're running the check-out action from the actions/checkout module. This action is used to checkout files to your local machine and update the contents of the file to match your repository's version. In our workflow, we want to check out the comments.txt file in our repository. Once the file is updated, we want to commit the file. We've also created a commit message and added a tag with the comment: "auto(comments): add comment"

If you're not familiar with YAML, it is just a data serialization language that is similar to JSON, but uses a lot less characters. After you have saved the YAML file, you need to make sure to save it as a file that has the .yml extension.

You can then trigger your GitHub Actions workflow with the following command:

```
curl -v -H "Accept: application/vnd.github.v3+json" \
    -H "Authorization: token $GITHUB_ACTIONS_TOKEN" \
    --request POST \
    --data '{"event_type": "newsletter-signup", "client_payload": {"comment": "Lara: This is nice I guess"}}' \
    https://api.github.com/repos/$USERNAME/$REPOSITORY/dispatches
```

Remember to replace $USERNAME and $REPOSITORY with the correct values. And you can [create a `$GITHUB_ACTIONS_TOKEN` here](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token).

We're using the HTTP POST method to trigger the event. We're sending the event_type and client_payload to the GitHub API. The client_payload is simply a JSON string. In the payload, we're adding a comment field with the value "Lara: This is nice I guess".

The response to this will contain the event name, but it will not tell you if the trigger has been successfully processed. So you will need to check the events in your repository's dashboard. Alternatively, you can also easily check the events in the repository using this command:

```
curl -v -H "Accept: application/vnd.github.v3+json" 
  -H "Authorization: token $GITHUB_ACTIONS_TOKEN" 
  --request GET 
  https://api.github.com/repos/$USERNAME/$REPOSITORY/events
```

To see if the trigger has been successfully processed, you can check out the repository events. Once you click on the event, it will show the status of the trigger. If the trigger has been successfully processed, the status will show as completed.

I'll leave it to you to figure out how to connect this to your frontend. My suggestion is to create an authenticated serverless endpoint, that receives POST events, massages the data, and POSTs again to GitHub.

I hope this has helped you understand more on using GitHub as your app's database. If you have any questions, feel free to reach out.


The world's slowest write database


To send me secrets, please encrypt them using [my GPG key](https://gunar.uk/gunar.pub.key).


Here's how to do it from the command line:

```sh
# Import my public key
curl -L https://gunar.uk/gunar.pub.key | gpg --import
# Encrypt secrets.txt using my key
gpg --encrypt --armor --recipient gunar@gunargessner.com <secrets.txt >encrypted.txt
```

Now the contents of `encrypted.txt` should look something like this:

```
-----BEGIN PGP MESSAGE-----
hQEMA27ixXDTzgLyAQf/ftCzhzDL2S1xGEh9zAYVVfSmyci1Os3EpxFTEtVEjCfk
dPbMoQ382/bA1fvM/h8Fu84bMqDi7qxWksU2GnC1pZk42wbOITtNtRv0S2M0SBEe
KkcU8PTL0tO7ePdez9O7nkyO3A8cnj3yrHMXjgpHK/ENZ3W6cV+pG5Tkzk976RFQ
-----END PGP MESSAGE-----
```

PS: For binary data (e.g. images or other larger files) please encrypt without the `--armor` argument so that it will generate a (smaller) binary output.


How to send me secrets


I recommend using [IIFEs](https://developer.mozilla.org/en-US/docs/Glossary/IIFE) instead of `let`s in the situation below.

Benefits:

- Immutable code
- Isomorphic to [do-expressions](https://github.com/tc39/proposal-do-expressions)


Before:

```js
let response;
try {
  response = parse(req.buffer);
} catch (e) {
  console.error(e);
}
```


After:

```js
const response = (() => {
  try {
    return parse(req.buffer);
  } catch (e) {
    console.error(e);
  }
})();
```


IIFE over let


```sh
# Display commands issued and exit script on first error
set -xe
# Log commands, stdout, and stderr to LOG_FILE
exec 3>&1 1>>${LOG_FILE} 2>&1
```


How to log all output from a bash script


See this [StackOverflow answer](https://softwareengineering.stackexchange.com/a/365344/316302).


No magic strings


I was not able to use [`got`](https://www.npmjs.com/package/got) on Firebase Functions.
Apparently `got` uses a feature (async generators) that is only available on newer version of Node.js.
This is the first time I have had a problem deploying serverless functions.
It never happens on [Vercel](https://vercel.com).
But I had to use Firebase Functions because I needed to react to an event in a Firebase Database.
Instead of playing whack-a-mole in trying to find a compatible library,
I just wrote one that requires zero dependencies.

## POST JSON

```js
const https = require("https");
async function post({ body, hostname, path }) {
  return new Promise((resolve, reject) => {
    const data = JSON.stringify(body);
    const options = {
      hostname,
      port: 443,
      path,
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Content-Length": data.length
      }
    };
    const req = https.request(options, res => {
      // console.log(`statusCode: ${res.statusCode}`);
      res.on("data", d => {
        // process.stdout.write(d)
      });
    });

    req.on("error", error => {
      console.error(error);
    });
    req.write(data);
    req.end();
    req.on("end", () => {
      resolve();
    });
  });
}
```


Make http/s requests with Node.js without any libraries


Hi there 👋 In this post we're going to leverage Hasura to seamlessly, automatically, and continuously sync a Postgres database into denormalized ElasticSearch documents.

## Why sync Postgres to ElasticSearch?

![Syncing Postgres to ElasticSearch](postgres-elasticsearch.png)

If *searching* is a core feature of our app, we will want to use ElasticSearch. Although [Postgres](https://www.postgresql.org/) does have a [Full Text Search](https://www.postgresql.org/docs/9.5/textsearch.html) feature, it doesn't scale well---especially when the requirements include searching across multiple tables.

[ElasticSearch](https://www.elastic.co/) is ubiquitous when we're talking about search engines. It is speculated that for a number of years [Lucene](https://lucene.apache.org/) (its underlying library) powered all of the Google search engine. ElasticSearch generates multiple indexes for each document inserted and that is how it manages to be so fast

So, how *do* we sync Postgres data to ElasticSearch ?

At the bottom of this post you'll find a number of open-source, self-hosted projects that have implemented solutions for this problem. Maybe you will like one of them. However, if we already have Hasura set up, the amount of code we actually have to write so little, that instead of adding another dependency to our application, we might prefer to write the glue-code ourselves. This is what we will be doing here.

## Pulling data from Postgres

![Setting up Hasura to observe database changes](postgres-triggers.png)

Hasura will be doing the heavy-lifting for us. It allows us to [trigger webhooks on database events](https://hasura.io/event-triggers) like INSERT, UPDATE, and DELETE. Hasura also manages the queue, handles errors and retries, and provides a web interface to inspect pending events and processed events. No need for Debeezium nor Kafka.

![Hasura event settings screen](hasura-event-settings.png)

You can find more information about Hasura Events at [Hasura Docs: Event Triggers](https://docs.hasura.io/1.0/graphql/manual/event-triggers/index.html).

## Transforming the data

![Transforming the received row into a denormalized document](transform-data.png)

Data stored in Postgres is relational in nature. Although ElasticSearch does implement the [Join Datatype](https://www.elastic.co/guide/en/elasticsearch/reference/current/parent-join.html), the documentation explicitly tell us that [de-normalization is key to good performance](https://www.elastic.co/guide/en/elasticsearch/reference/current/parent-join.html#_parent_join_and_performance).

### What is denormalization?

For the purposes of the present article we can think of denormalization as nothing more than just "expanding IDs into full blown objects."

![Normalization vs Denormalization](denormalization.png)

### How to denormalize the data?

![Using GraphQL to denormalize data](use-graphql-to-denormalize-data.png)

We could use [SQL JOINs](https://www.postgresql.org/docs/9.3/tutorial-join.html) in order to denormalize the data, but you know what else is great at denormalizing stuff? GraphQL! When you think of it, denormalization is one of the biggest benefits of GraphQL. It's a core feature of the protocol. And since we already have Hasura set up, we get our GraphQL API for free.

```graphql
query MyQuery($id: uuid!) {
  purchases_by_pk(id: $id) {
    id
    customer {
      name
      phone
    }
  }
}
```
## Pushing the transformed data to ElasticSearch

![](push-to-elastic.png)

The response from the GraphQL API is in the JSON format, which means we can push it directly into ElasticSearch.

ElasticSeach's API is a  REST API, so we can use the `PUT` verb for row insertions and updates, and the `DELETE` verb for row deletions. Here's one example:

```
PUT http://localhost:9200/purchases/_doc/1

{
  customer: {
    name: 'Steve Jobs',
    phone: '+1 555 555',
  }
}
```

## Conclusion

![](complete-flow.png)

There we go, the plan for a complete one-way sync from Postgres to ElasticSearch. Here we have not written any actual code here---as the code would depend on your specific data structure and business needs---but my experience is that we would need to write just a little bit of glue code.

Hasura reminds me of a Swiss Army knife. Once in our stack, it allows us to insert, mutate, query, and stream data in a myriad of ways. Looking forward to hearing about your experience in the comments.

---

## Prior work

-   [zombodb](https://github.com/zombodb/zombodb) is a postgres extension that implements custom indexes. Updates to tables indexed by zombo trigger calls to the ElasticSearch API to update documents. Zombodb also allows us to run ElasticSearch queries from within SQL-land.
-   [pg-sync](https://github.com/toluaina/pg-sync) is a python library that not only captures data changes but also allow us to use JSON configurations to transform the data (e.g. denormalizing).
-   [abc](https://github.com/appbaseio/abc#abc) is a CLI tool written in go, built by appbase.io, which *can *be used to sync Postgres/SQL/Mongo to ElasticSearch.

*NB: Out of the options above, only pg-sync gives us an elegant API with which to denormalize our data.*


Sync Postgres and ElasticSearch using Hasura today


This is a GitHub Action template for force-pushing one repo into another once every day.

1. [Create a new repo](https://github.new) called `repo-sync`
2. [Create a Personal Access Token for GitHub](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token)
3. [Add the token as a secret](https://docs.github.com/en/actions/reference/encrypted-secrets) called `PERSONAL_ACCESS_TOKEN`
4. Put the code below into `.github/workflows/sync.yml` and replace the `<variables>`
5. Profit (i.e. you're done!)

```yaml
name: "Daily sync"

on:
  push:
    branches:
      - main
  schedule:
    -   cron: "0 0 * * *"

jobs:
  backup:
    runs-on: ubuntu-latest
    name: Sync repos
    steps:
    - name: sync
      env:
        PERSONAL_ACCESS_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
        USERNAME: <your-username>
        REPO_FROM: <org/repo>
        REPO_TO: <org/repo>
      run: |
        set -xe
        git clone "https://${USERNAME}:${PERSONAL_ACCESS_TOKEN}@github.com/${REPO_FROM}.git" .
        git remote add fork "https://{$USERNAME}:${PERSONAL_ACCESS_TOKEN}@github.com/${REPO_TO}.git" 
        git fetch fork
        git push --force fork origin/main:main
```


Keep two repositories in sync


Wealth comes from profits.
But where does profit come from?

Before we answer that, we need to know what profit is.

## What is profit?

If you ask anyone on the street what profit is,
they will give you the definition of **Accounting Profit**,
which reflects the formula

```
profit = revenue - cost
```

There's also another profit, called **Economic Profit**, and has the formula

```
profit = revenue - financial cost - opportunity cost
```

But what is the opportunity cost variable, and why does it matter?

Imagine we owned a piece of land,
which can grow onions and corn.
For the current season,
we must decide what to plan on this land.
How to make that decision?
We would run simulations, or projections,
for each of these vegetables.
We would estimate the costs
and the price at which we would able to sell them later.
We can't grow both on the same squared meter,
although it should be possible to plant both
if we were to divide our land in sections.
Still, we would need to decide the ratio
of squared meters to plant
onions and corn.
If we plant corn we can't plant onions.
If we plant onions we can't plant corn.
This is the opportunity cost.

It's possible to have accounting profit but not economic profit.
Let's say we planted corn.
Come harvest time, the sell the corn for a monetary amount
greater than the cost of planting it.
That means we have accounting profit, where `revenue > cost`.
This is great, right? Well, maybe.
We talk to our neighbors, who had planted onions instead,
and learn that with the same amount of land,
they have made double the revenue that we have.
Although making accounting profit is good, it's not great.
What is great is to make economic profit.
In this case, it would have meant to have planted corn instead.

## How to make Economic Profit

Economic profit relates to optimizing the use of limited resources.
What are the resources we, Software Engineers, have at our disposal?
Our main resource is obviously time.
Time is as valuable as it's limited.

As we have seen above, to optimize the use of our limited resource,
we need to run simulations or projections.

Here are some scenarios I have prepared for us.
The constant here is the resource.
It assumes an investment of 40h/wk of our limited resource time.

The scenarios are:

1. Getting a job paying $130k/year
2. Working as consultants charging $200k/year
3. Building a product that generates revenue


![Chart of revenue over time.](chart.png)

We can see that working in a job, in our industry, pays well.
Starting our own consultancy company pays even better.
Now, why is the Product line a curve?

This simulation shows us that the market is not fair.
Profit is an event that may or may not happen.
Hence, a whole year of hard work with nothing to show for.
On the second year, things start picking up.
Still, how come have we reached \$1MI in the following year?

Remember, economic profit comes from optimizing the use of our limited resources.
In our case, time.
Working at a job or as a consultant gives us revenue proportional to the investment.
That is, the ratio of time invested to revenue is constant.
That is not the case with digital products.

How much more time do we need to invest to sell our product to our 1,001th customer?
Certainly less time that it took us our 1st.
Hence, the ratio changes.
We start being able to invest less time for the same amount of revenue.
In the scenario above, we are using this spare time
double-down our investment into our product,
looking to increase the angle of this exponential curve.

We need to be aware of a product's life cycle.
Nothing lasts forever.
But after building the first,
we'll have time, money, and know-how to build the next one.


How to get rich as a software engineer


_This is an old blog post.  I'm keeping it alive for *archiological reasons only*.  [Check some recent stuff instead](/)._

The JavaScript community has little agreement on how to Subclass Exceptions
(a.k.a. Subclass Error, Extend Error, Custom Errors).
At the root of the problem lies the fact that JS has Prototypical Inheritance,
not Classical Inheritance.

I take for granted here that Classical Inheritance is problematic and one better avoid
it. Also that \`instanceof\` "lies".

You can find resources on the subject here:

- [Common Misconceptions About Inheritance in JavaScript](https://medium.com/javascript-scene/common-misconceptions-about-inheritance-in-javascript-d5d9bab29b0a)
- [What’s the Difference Between Class & Prototypal
    Inheritance?](https://medium.com/javascript-scene/master-the-javascript-interview-what-s-the-difference-between-class-prototypal-inheritance-e4cd0a7562e9)

# Custom Errors with Duck-typing

Let's say our code is expected to throw errors with messages the users should be
aware of. Here is what I would do:

```js
'use strict'

const MESSAGE_FOR_USER = Symbol()
const userError = function (message) {
  const error = Error(message)
  Object.defineProperty(error, MESSAGE_FOR_USER, { get() { return message } })
  // Omit frames above userError
  if (Error.captureStackTrace) Error.captureStackTrace(error, userError)
  return error
}
const getMessage = e => e[MESSAGE_FOR_USER] 

// -----

function foo() {
  throw userError('User be aware: Foo has been called.')
}

// Dummy function to examplify sending a message to the user
const send = msg => console.log(msg)

try { foo() }
catch (e) {
  const msg = getMessage(e)
  if (msg) { send(msg) } 
  else {
    // Unexpected exception
    console.error(e)
  }
}

// User be aware: Foo has been called.
```

Let's break it down...

## Symbol
We need to store the message in our error object.
We can put it under a key named \`message\` or \`userMessage\`.
However "message" is a common word and there's the possibility of conflicts
with other code touching the same keys.

[Symbol][], a ES2016 feature, allow us to create unique identifiers.
We can use a symbol to create a unique key in the object.
For example

```js
const UNIQUE_KEY = Symbol('anything, really')
const object = {}
object[UNIQUE_KEY] = 'Data'
```

.

It can be further simplified with ES2015's [Computed Property Name][],
becoming

```js
const UNIQUE_KEY = Symbol('anything, really')
const object = { [UNIQUE_KEY]: 'data' }
```
.

## Putting it under a Getter

Defining the key as a [getter]() gives us confidence that no code is going to mutate
that value.

One way to define a \`getter\` is with \`defineProperty\`, thus

```js
  Object.defineProperty(error, MESSAGE_FOR_USER, {
    get() { return message },
  })
```

.

### Side node: Getter
We could have set the \`getter\` at object creation,

```js
const error = {
  get [MESSAGE_FOR_USER]() { return message }
}
```

, however in this case we can't because we want to build our object off of \`Error\`.

## Error.captureStackTrace
\`userError()\` creates an \`Error\` object to be used as base for our error.
This object has a \`stack\` property (i.e. \`error.stack\`).
Stack-trace information is useful so we can see where the error has been thrown.

Here \`error.stack\` contains information about the place where the error object
was created.
We don't want that. We only want the stack from where the object has been
thrown.
For example:

```js
'use strict'

function e() {
  const error = Error()
  return error
}
throw e()
```

outputs

```
Error
    at e (code/subclassing-exception.js:4:17)               
    at Object.<anonymous> (code/subclassing-exception.js:7:7)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.runMain (module.js:605:10)
    at run (bootstrap_node.js:425:7)
    at startup (bootstrap_node.js:146:9)
```

The line that refers to \`throw\` has been pushed to second place.
The first line points to \`e()\`.
This is not such a big of a deal, as we could always just ignore that first
line.

We could mutate \`error.stack\` as a string. However \`Error\` is implementation-specific
and our code could break in different environments.


In \`Node.js\`, there's a helper function called \`Error.captureStackTrace\` which
does exactly that for us. For example

```js
'use strict'

function e() {
  const error = Error()
  Error.captureStackTrace(error, e)
  return error
}
throw e()
```

outputs

```
Error
    at Object.<anonymous> (/code/subclassing-exception.js:8:7)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.runMain (module.js:605:10)
    at run (bootstrap_node.js:425:7)
    at startup (bootstrap_node.js:146:9)
    at bootstrap_node.js:540:3
```

No reference to \`e()\`.

# Spice it up
I like my JS with splashes of \`lodash/fp\` and \`modules\` (in \`Node\`).
So what I would actually do is

```js
'use strict'

const { get } = require('lodash/fp')

const MESSAGE_FOR_USER = Symbol('userError')

const userError = message => {
  const error = Error(message)
  Object.defineProperty(error, MESSAGE_FOR_USER, { get() { return message } })
  // Omit frames above userError
  if (Error.captureStackTrace) Error.captureStackTrace(error, userError)
  return error
}

const getMessage = get(MESSAGE_FOR_USER)

Object.assign(userError, {
  getMessage,
})

module.exports = userError
```

```js
'use strict'

const userError = require('./userError')

function foo() {
  throw userError('User be aware: Foo has been called.')
}

// Dummy function to examplify sending a message to the user
const send = msg => console.log(msg)

try { foo() }
catch (e) {
  const msg = userError.getMessage(e)
  if (msg) { send(msg) } 
  else {
    // Unexpected exception
    console.error(e)
  }
}

// User be aware: Foo has been called.
```

Thank you.


[Symbol]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol
[Computed Property Name]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Object_initializer#Computed_property_names
[getter]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/get


There is no "Subclass Exception" in JavaScript


_tl;dr Use [cliget](https://github.com/zaidka/cliget/) to fetch the URLs, then on a VPS pipe `curl` into `aws s3 cp`._

After [numerous posts on HackerNews](https://hn.algolia.com/?q=google+account) about Google locking people out of their accounts, I've decided to take initial steps towards securing my data.

[Google Takeout](https://takeout.google.com/settings/takeout/) allows you to download all your data. I don't remember the specifics but I'm pretty sure they were forced to offer it to comply with GDPR. The downloaded archive can contain your GDrive, GMail, Google Photos, and a whole bunch of other stuff you've probably forgotten you even had.

But where to store these huge files?

## Costs

My own Google Takeout archive is 400GB big. It costs me $0.40/mo in AWS storage and would cost me $35 if I ever needed to download it.

Downloading is expensive because it incurs [outbound data transfer fees](https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer). It's cheap if you consider that I hope—knock-on-wood—to never need to access this backup.

Finally, the _process_ I've used to download the data and upload to AWS requires a VPS, which has cost me a one time fee of $0.40.



You can run your own estimates using the [AWS Calculator](https://calculator.aws/#/createCalculator/S3).

### Glacier vs Deep Archive

Should you use Glacier or Deep Archive? You'll notice the fee of early deletion is 2x larger for Deep Archive but this difference is dwarfed by the 75% storage price discount. Deep Archive is always cheaper.

![Comparison of Glacier vs Deep Archive](glacier-vs-deep-archive.png)

---

I'll walk you through the process of exporting a backup from Google Takeout and uploading it to AWS S3 Glacier.

### What you'll need

- [An AWS S3 Bucket](https://aws.amazon.com/s3/)
- [An AWS EC2 server](https://aws.amazon.com/ec2/) (I suggest the smallest one the [t4g.nano](https://aws.amazon.com/ec2/instance-types/t4/))
- [Firefox](https://www.mozilla.org/firefox) with [the cliget extension](https://github.com/zaidka/cliget/)

## 1. Request a Google Takeout archive

Go to [Google Takeout](https://takeout.google.com/settings/takeout/) and create a new export.

I strongly suggest you pick the file type `.tgz`, so that you can set the size of each part to be `50GB`. The more files there are, the longer this process will take.

![Delivery method](takeout-delivery-method.png)

## 2. Prepare your VPS

In theory you can avoid renting a VPS from EC2 by using [AWS CloudShell](https://aws.amazon.com/cloudshell/) directly.
In practice it doesn't work because the [CloudShell will automatically close after 20 minutes of inactivity](https://docs.aws.amazon.com/cloudshell/latest/userguide/limits.html#session-lifecycle-limitations)
and transferring 50GB .tgz files form Google Takeout takes more than 1 hour.

Although you could rent a VPS elsewhere (e.g. Google Cloud or Vultr),
it's ergonomic to have the server in the same data center as the S3 bucket.

Create an EC2 instance (the [t4g.nano](https://aws.amazon.com/ec2/instance-types/t4/) is enough) and ssh into it.  Copy the script below and replace `<your-bucket-name>` with—guess what—your bucket name.

If you're going to encrypt your data on the client-side (I would), then replace `<your-email-address>` with your GPG key's email address. Also make sure to import the key with `gpg --import`.

Otherwise, just remove the `gpg` line below.

### `dl.sh`

```sh
##!/usr/bin/env bash

## Usage: ./dl.sh curl …

## XXX: Set your bucket name here
AWS_S3_BUCKET="<your-bucket-name>"

## XXX: Set your gpg encryption email address here (or remove the `gpg` line)
GPG_EMAIL="<your-email>"

## Gets last argument
filename=${*:16:1}

## Gets all but the last 3 arguments
args=("${@:2:$#-3}")

echo "Downloading: $filename"

## Expands all args (i.e. `args[@]`) in a format that can be reused as input (i.e. `@Q`)
curl "${args[@]@Q}" \
  | gpg --encrypt -r ${GPG_EMAIL} --trust-model always  \
  | aws s3 cp - "s3://${AWS_S3_BUCKET}/${filename}" \
    --storage-class DEEP_ARCHIVE 
```

### Optional: Create a swapfile

During my experiments with a [t4g.nano](https://aws.amazon.com/ec2/instance-types/t4/), 0.5GB of RAM seems enough,
but I recommend creating a swapfile just in case.
It helps to prevent the ssh session becoming unresponsive.


Here's a working snippet that creates a 1GB swapfile:

```bash
sudo dd if=/dev/zero of=/swapfile count=1024 bs=1M
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo "/swapfile   none    swap    sw    0   0" | sudo tee -a /etc/fstab
```

## 3. Generate download URLs

Google Takeout archives can be downloaded from a browser only.
Google doesn't offer a URL that works with `curl`.
Of course you can use your browser and download them all normally, but you'd be limited by your
computers storage and most importantly by your home internet connection—which is required twice, once for downloading from Google and once to upload to AWS.

The only solution I've found is to use the Firefox extension called [cliget](https://github.com/zaidka/cliget/).
This extension allows you to copy the URL of a download—including headers and cookies—_before_ the download starts.

I've looked for Chromium-based alternatives but it doesn't seem like there are any.
If Firefox is not your main browser, you can always install it just for the sake of this process.

Wait for email confirmation that your takeout is ready.

![email from Google](email.png)

Spin up Firefox and navigate to [Google Takeout](https://takeout.google.com/settings/takeout/).
Click to download the first part. When the Download modal opens, dismiss it and click on the extension icon instead.

![cliget](cliget.png)

Copy this text, we're going to use it in the next step.

![cliget](cliget-2.png)

## 4. Transfer the files

On your VPS (ssh), call [`dl.sh`](#dlsh) and use as arguments the command line copied from cliget—starting with `curl` and ending with `--output` — e.g. `./dl.sh curl …`. Do this for each part.

You'll notice the download slows down after a while. Starts at 13MB/s and slows down to less than 1MB/s.
I've checked CPU and RAM usage, and even checked the [network bandwidth](https://www.speedtest.net/result/c/7cf9157c-7d11-42dc-a39b-7e4697668b2c) but couldn't find the bottleneck. I'm pretty sure it's just Google's servers throttling the downloads.

## 5. All done

After kicking off downloads for each file part, and waiting a few hours for it to complete, I'd managed to have the whole 400GB of my Google Takeout stored in AWS Deep Archive for $0.40/mo.

![s3](s3.png)

Friendly reminder. If you're on GMail, please set up a custom domain. It'll allow you to forward your emails to a different provider should the worst ever happen.


Storing Google Takeout on AWS Glacier


Tests are usually made of three different concerns:

1. Setup
2. Execution
3. Assertions

When writing tests, you should represent this distiction visually, using empty lines:

```js
test("returns foo", () => {
  // Setup

  // Execution

  // Assertions
})
```

Here's an example:

```js
describe("verifyKey", () => {
  it("returns true for valid cryptographic keys", () => {
    const { publicKey } = crypto.generateKeyPair();

    const actual = crypto.verifyKey(publicKey);

    const expected = true,
    assert.strictEqual(actual, expected);
  });
});
```


Testing: Separation of concerns


```js
function useSt8(initial) {
  return st8.bind(useState(initial));

  function st8() {
    switch (arguments.length) {
      case 0:
        return this[0];
      case 1:
        return void this[1](arguments[0]);
      default:
        throw new Error("Expected 0 or 1 arguments");
    }
  }
}
```

[Originally seen here](https://github.com/mweststrate/use-st8), but posted here as a snippet because it's small enough to just copy and paste instead of growing the dependency tree.


Single-function alternative for React.useState


I disencourage returning values that aren't supposed to be returned:

```js
  // Don't do this
  onCopy={() => setCopied(true)}
```

The function above returns the result of `setCopied()`, which in the case of React is most likely `undefined` and shouldn't create problems. However, what if the code behind `onCopy` changes to expect certain things to be returned?

Better to only return expected values:

```js
  onCopy={() => {
    setCopied(true);
  }}
```

The code above explicitly returns `undefined`, even if `setCopied` ever changes.

And you can get rid of the curly braces by using the `void` keyword:


```js
  onCopy={() => void setCopied(true)}
```




void over curly braces


```
lame -V 9 -h input.wav output.mp3
```

See `man lame` for more options.


How to convert WAV to MP3 using a Linux CLI


In October [we](https://inv.tech) decided to make a horizontal investment in speed and decided to go monorepo with our services architecture. Naturally, we still want to be deploying to microservices for scalability and resilience — so it’s _not a monolith, it’s a monorepo_. And I was entrusted with the implementation.

---

## The benefits

There are quite a few [benefits to the monorepo architecture](https://gomonorepo.org/), and three of them stood out for us.

### 1. Code review

Code changes to multiple different microservices can be contained in a single pull request. So you have the full scope of the change in a glimpse. You can even release new versions of packages and update the services that consume them at the same time.

### 2. End-to-end testing

We must admit, in favor of being nimble we’ve been skipping writing E2E tests, but now for DAL2 we’re ready to make the commitment. We want to write Unit Tests, then E2E Tests, and only write Integration Tests where absolutely necessary (because in my experience, the latter carries the highest maintenance costs).

The monorepo will simplify this process. There are a couple of ways we can implement them in the monorepo architecture, and we’re still working on it, so I’ll leave it another report (next month?).

### 3. One deployment per Pull Request

This is the (my) holy grail. I want us to be able to develop locally, push changes to the cloud, and have our infrastructure spin up the whole thing. Imagine working on a feature and being able to provide to the stakeholders with a unique URL for them to test, knowing that (1) that’s exactly how the system will behave once merged into production, and that (2) they can fiddle around as much as they want, as this environment is not shared with production nor a centralized staging environment. We’re not there yet, but that’s where we want to get.

---

## The setup

### File structure

We’re using [lernajs](https://github.com/lerna/lerna) alongside [yarn workspaces](https://yarnpkg.com/lang/en/docs/workspaces/) to manage dependency installation and linking, and running tests for every service.

![So far we have ported to the monorepo 5 services and 1 package.](tree.png)



### CI Workflow

We’re using CircleCI for testing and deployment. We use a custom `.circleci/config.yml` file, that calls lerna for tests, and calls a custom bash file for deployment (scripts/deploy.sh).

![A successful `deploy` workflow. Note that services are deployed in parallel.](ci.png)

### Dockerfile

We’re basing our image on the [Alpine](https://www.alpinelinux.org/) disto. Then doing a little `PATH` magic, and using `yarn` for dependencies.

```Dockerfile
FROM alpine:latest
WORKDIR /usr/src/app
RUN apk add --no-cache --update 'nodejs~8' yarn
RUN yarn global add node-gyp
RUN export PATH=$PATH:./node_modules/.bin
# http://bitjudo.com/blog/2014/03/13/building-efficient-dockerfiles-node-dot-js/
ADD package.json yarn.lock ./
RUN yarn install -s --ignore-scripts
ADD . .
RUN yarn build
CMD ["yarn", "start"]
```

### Dockerhub

Our deploy script builds the docker image on the cloud (CircleCI) then pushes the images to Dockerhub. CircleCI has their own Docker Caching Layer so it skips redundant builds. Good job, CircleCI!

![All image names were prefixed with `ygg` to avoid conflicts and overwrites during the transition.](dockerhub.png)

### Heroku

We host our code on Heroku, and we were delighted to learn that Heroku has their own Docker Registry, so pushing an image is as simple as `docker push registry.heroku.com/<repo>/<service>:<tag>`.

![The end result. It's deployed!](heroku.png)

To level with you, after pushing the image, you still have to release it. Here’s the code

```sh
#!/bin/sh

http_code=$(curl -s -o out.json -w '%{http_code}' -n -X PATCH https://api.heroku.com/apps/$HEROKU_APP_NAME/formation \
  -d "{
  \"updates\": [
    {
      \"type\": \"web\",
      \"docker_image\": \"$IMAGE_ID\"
    }
  ]
}" \
  -H "Content-Type: application/json" \
  -H "Accept: application/vnd.heroku+json; version=3.docker-releases" \
  -H "Authorization: Bearer $HEROKU_API_KEY" \
  ;)

# Display output on the screen
cat out.json

if [[ $http_code -eq 200 ]]; then
  echo "****** Image successfully relased (Heroku)"
  exit 0
fi
echo "****** There was an error when trying to release the image"
exit 1
```

---

## Reviewing the first PR to the Monorepo

In order to send this PR for review I was careful to keep the git history as clean as possible. As this is a new repo, I wanted master to be clean and everything to be on the first PR. This meant a lot of rebasing :)

![Can you guess what “Batman” stands for?](batman.png)

![Merging existing repos into the monorepo while dealing with open PRs was tricky, so these commit messages are not consistent with semantic types, but will be, moving forward.](commits.png)

And I even learned [how to split a commit in two](https://twitter.com/gunar/status/1058066168207032320) while cleaning up the history, which I readily shared with the team (and with Twitter).

```sh
git rebase -i <oldsha1>
# mark the expected commit as `edit` (i.e. replace pick in front of the line), save and close
git reset HEAD^
git add .
git commit -m "First part"
git add .
git commig -m "Second part"
git rebase --continue
```

---

I’m stoked about the peace of mind the monorepo architecture provides to my team and me, and what it will yield us in terms of speed and quality.


Gunar Gessner

Newsletter

Writings (rss)