chriscolborne.com

Don't use method syntax for your TypeScript function types!

Thu, 31 Mar 2022 10:43:29 GMT

In TypeScript, there are a few ways to type functions. But did you know that using the method syntax to define your functions in TypeScript is not type safe?

Typing Functions

There are three main ways you can type a function in TypeScript.

1. Function type expression syntax

The most common and recommended way is to type it as a function type expression, which uses a syntax like an arrow function:

interface MyObj {
  add: (a: number, b: number) => number;
}

2. Call signature syntax

Another less common way is as a call signature. This is often useful if the function has properties as well as being callable.

interface MyObj {
  add: {
    description: string;
    (a: number, b: number): number;
  };
}

3. Method syntax

The final way is method syntax. This often makes sense when talking about classes but is also valid for interfaces.

class MyCalculator {
  add(a: number, b: number): number;
}

interface MyObj {
  add(a: number, b: number): number;
}

But did you know if you use the method signature syntax (#3 above), it’s actually not completely typesafe.

strictFunctionTypes flag in TSConfig

The strictFunctionTypes flag in TSConfig, when enabled (as it is with strict: true), causes functions parameters to be checked more correctly.

However from the docs:

During development of this feature, we discovered a large number of inherently unsafe class hierarchies, including some in the DOM. Because of this, the setting only applies to functions written in function syntax, not to those in method syntax.

That means, if you type your function with the method syntax, strictFunctionTypes does not apply! 🙀

Let’s see an example.

interface PrintMethod {
  print(a: string | number): string;
}

interface PrintFunction {
  print: (a: string | number) => string;
}

interface PrintCall {
  print: {
    (a: string | number): string;
  };
}

const printFunc = (a: string) => 'hello';

const MyAddingMethod: PrintMethod = {
  print: printFunc, // uh oh - this isn't flagged as an error
};

const MyAddingFunction: PrintFunction = {
  print: printFunc, // this is correctly flagged
};

const MyAddingCall: PrintCall = {
  print: printFunc, // this is correctly flagged
};

TypeScript Playground

It’s not all bad news - it will correctly catch it if the type changes completely, for instance from a `string` to a `number`, however I’d argue that that makes it even more insidious, sometimes it catches it, sometimes it doesn’t.

What should you use?

Long story short, you should avoid using the method syntax (even for classes) when typing your functions in TypeScript. Use the function syntax (or the call signature if you need the additional properties) and your code will be much type-safer for it!

How can I run some code after changing state with React hooks?

Sat, 05 Dec 2020 14:49:14 GMT

Running some code after changing state is a common desire. Let’s look at how to do that with hooks.

Method 1 - useEffect hook

The first and most commonly used method to run a function after updating state is the useEffect hook. useEffect runs its function only when the items in the dependency array change. Sounds pretty spot on for what we want huh? Let’s see it in action:

const [name, setName] = useState('John Doe');
useEffect(() => {
  // run something every time name changes
  console.log(name);
}, [name]); // <-- dependency array

Pretty simple, every time name changes, we’ll run the function, which console.logs the name out.

Method 2 - useReducer hook

When you find yourself needing more complex state updates, where updating one item of state affects another item of state, you can reach for useReducer.

What if we had a status, and wanted to save a history of the last 10 statuses? Because the history relies on changes to the status, useReducer would be a good choice here. Let’s see it in action:

const CHANGE_STATE = 'CHANGE_STATE';
const CLEAR_HISTORY = 'CLEAR_HISTORY';

const reducer = (state, action) => {
  const { status, history } = state;
  const { type, newStatus } = action;

  switch (type) {
    case CHANGE_STATE:
      let newHistory = [...history, status];
      if (newHistory.length > 10) {
        newHistory.shift();
      }
      return { status: newStatus, history: newHistory };
    case CLEAR_HISTORY:
      return { status, history: [] };
    default:
      return state;
      break;
  }
};

export const StatusHistory = () => {
  const [state, dispatch] = useReducer(reducer, {
    status: 'stopped',
    history: []
  });
  const start = () => dispatch({ type: CHANGE_STATE, newStatus: 'started' });
  const finish = () => dispatch({ type: CHANGE_STATE, newStatus: 'finished' });
  const clear = () => dispatch({ type: CLEAR_HISTORY });

  return (
    <div>
      <p>Current status: {state.status}</p>
      <ul>
        {state.history.map((previousStatus) => (
          <li>{previousStatus}</li>
        ))}
      </ul>
      <button onClick={start}>Start it</button>
      <button onClick={finish}>Finish it</button>
      <button onClick={clear}>Clear History</button>
    </div>
  );
};

Hopefully, this has shown you two options for running code after changing state with React hooks.

tl;dr - if the state (and code to run) is simple, you can go with useEffect. If you’ve got multiple items of state that need to be updated based on changes, go with useReducer.

Object Destructuring 101

Sun, 08 Nov 2020 00:00:00 GMT

aka what on Earth does `doIt({ x: x2 = getX() } = {})` mean?!

You’re getting used to React. You’ve got components and arrow functions down pat. But then you run into this beast:

function doRender (
  ui,
  {
    state: initialState,
    options = getOptions({ overrides })
  } = {}
) { // ... }

Wait, what, how? Brackets and colons and equals, oh my! This little snippet breaks your brain. What was this developer thinking? Were they just being too clever, or what?!

Although it certainly is bracket soup, there is a method to the madness.

This is a few different levels of object destructuring, layered on top of each other, inside a function call. Let’s break it down bit by bit, so that you can read it like a pro.

Level 1: Basic Destructuring

First up, let’s start with the basics. Object destructuring is just a way of extracting certain keys directly from an object. It’s used quite heavily in React and other modern JavaScript frameworks. In fact you probably already use it. It looks like this in its basic form.

const myObject = {
  name: 'Chris',
  email: 'chris@example.com',
  city: 'Brisbane',
};

// extracts 'Brisbane' and assigns it to a variable `city`
const { city } = myObject;

Easy right? Let’s keep going.

Level 2: Renaming Destructuring

So next level, what if we already had a variable city? Let’s rename it as we extract it:

const myObject = {
  name: 'Chris',
  email: 'chris@example.com',
  city: 'Brisbane',
};

// oops we already have city in scope
const city = 'Sydney';

// extracts 'Brisbane' and assigns it to a variable `myCity`
const { city: myCity } = myObject;

Two from two, got it.

Level 3: Multi-Level Destructuring

Next up let’s tackle multi-level destructuring. That’s when the variable you want to destructure is actually nested inside another key. Let’s try and get at city and state in this nested object.

const myObject = {
  name: 'Chris',
  email: 'chris@example.com',
  address: {
    city: 'Brisbane',
    state: 'QLD',
  },
};

// now city variable is 'Brisbane' and state variable is 'QLD'
const {
  address: { city, state },
} = myObject;

Notice a trick here - address isn’t actually destructured, it’s just used to get at its children. If you wanted the full address as well, you could either destructure the address first, then destructure address into city and state, or destructure twice.

// destructure `address` then `city` from `address`
const { address } = myObject;
const { city } = address;

// destructure `address` itself, then `city` from within it
const {
  address,
  address: { city },
} = myObject;

Great, we’re starting to look like that initial snippet.

Level 4: Destructuring Defaults

Next level is destructuring defaults. Up until now, we’ve been assuming the data is there. But what happens if a particular key might be there, or might not? That’s where defaults come into play.

const myObject = {
  name: 'Chris',
  email: 'chris@example.com',
  // city is missing for this one
};

// `city` in this case will be `undefined`
let { city } = myObject;

// let's add a default
// now `city` will be 'Sydney' since it's not set in `myObject`
let { city = 'Sydney' } = myObject;

const myObject2 = {
  city2: 'Brisbane',
};
// but `city2` here will be 'Brisbane' since it was set in `myObject2`
const { city2 = 'Sydney' } = myObject2;

When we try to do multi-level destructuring (or more generally try to destructure something that might be undefined), that’s where we might run into problems. Take this example, we try and get the city from the address, but there is no address in myObject.

const myObject = {
  name: 'Chris',
  email: 'chris@example.com',
  // sometimes we have address, but not this time
  // address: {
  //   city: 'Brisbane',
  // }
};

// bah bow - cannot read property 'city' of undefined
const {
  address: { city },
} = myObject;

// so let's fix that with a default empty object
// now we're looking for `city` in an empty object,
// which won't fail - `city` will be undefined
// but won't blow up
const { address: { city } = {} } = myObject;

Full Circle

So now we’re back to our original brain breaker. We can see now that all we’ve got is some multi-level destructuring with defaults.

Still not convinced? Ok, we’ll step through it bit by bit to make sure it sinks in:

// function call
function doRender (

  // first parameter called `ui`
  ui,

  // destructure the second parameter
  {

    // extract and rename `state` to variable `initialState`
    state: initialState,

    // extract `options` to a variable, and if it's unset,
    // default to the result of `getOptions()`
    options = getOptions({ overrides })

    // finally, default the second parameter to empty object, as it is optional
  } = {}

) { // ... }

Hopefully, this has helped you see that even the most confusing looking destructuring is made up of these 4 levels. Parse them one by one, and you’ll be reading and writing code like this in no time.

Which Headless CMS should I use for a new Gatsby site?

Sun, 19 Apr 2020 21:00:00 GMT

Gatsby is an awesome static site generator, but you need to get your data from somewhere. What are your options for headless CMSs?

What is Gatsby?

Gatsby is a free and open source framework based on React that helps developers build blazing fast websites and apps. It basically bashes together React, GraphQL and the so called JAM Stack to make some serious awesomeness.

What is the JAM Stack?

The JAM Stack is a name standing for a modern way of building sites/apps with JavaScript, APIs and Markup. Commonly it involves server side generated code from Markdown and/or API endpoints.

What is a headless CMS?

In this new JAM Stack world, your CMS backend and your frontend site are split. A headless CMS is simply a CMS which instead of pumping out HTML and CSS views, produces an API. There are many options from open source, to paid SaaS options, to trusty old WordPress.

What are the options?

Open source options

Netlify CMS

Netlify is one of the most common hosting providers for the JAM stack. They’ve built an open source SPA which you can install as an npm module on your site, to provide a CMS experience over GitHub and Markdown, complete with editorial workflow using pull requests.

Try it out

Strapi

Strapi is an open source, self hosted headless CMS written in Node. It provides both REST and GraphQL endpoints, with a customisable admin portal and API.

Try it out

Keystone

Keystone is a scalable, extensible and open-source platform to build NodeJS applications. It has first-class GraphQL support, and a great Admin UI.

Try it out

WordPress

Even though WordPress is obviously a full CMS and blogging platform, it can also be used as a headless CMS by using its REST API or using the WPGraphQL plugin to provide a GraphQL endpoint. One thing to note is that you may have more or less success depending on how many plugins you use.

Try it out

Software as a Service (SaaS)

Sanity

Sanity is one of the new breed of SaaS headless CMSs. It consists of an open source CMS admin UI which you install as an npm package and host. This allows you to easily extend for your own workflow with their extension ecosystem. The admin then connects to Sanity for storing the data, and for authentication.

Try it out | Pricing

Contentful

Contentful is one of the larger SaaS headless CMSs, and the oldest on our list here, being launched back in 2014. Clearly pitching themselves as the Enterprise solution, Contentful has the most clout, but also the price point to match. One thing to watch is that they may not have the features and extensibility common in the current generation of SaaS headless CMSs.

Try it out | Pricing

Prismic

Prismic is another of the new breed of SaaS headless CMSs. It provides an excellent hosted admin with the ability to easily add custom types, and publishes a REST and GraphQL API.

Try it out | Pricing

Forestry

Forestry has been described as Netlify CMS on steroids. Like Netlify CMS, it connects to GitHub to store markdown files which is great for portability. Unlike Netlify CMS however, Forestry also provides a hosted CMS admin UI, providing a more polished, managed solution. However you can also deploy the editor to your own site with Forestry Remote Admin.

Try it out | Pricing

Which one should you use?

Like everything, it depends! 🤷‍♂️

A lot of it comes down to how you want to store your data (git / hosted service) and the editing experience.

If it’s a commercial site, you’ve got the money, or just really want something turnkey, then try one of the SaaS options. They all have free plans to start, so sign up and try them out to see what fits.

If it’s a simple blog, Netlify CMS will do great. (in fact, I’m using it for this blog!)

If it’s a bigger site / application, give a more full powered solution a go such as Keystone or Strapi.

And if you need to work with clients, you might be stuck with WordPress, but at least you can have some shiny front end tech and they get the benefits of a performant, secure site.

Can I use React Helmet to add social card meta tags for Twitter and Facebook?

Tue, 18 Feb 2020 08:30:00 GMT

So you’ve built an awesome React powered site. It’s shiny, and cool, and does everything you need… until your client/boss/best mate’s dad asks about social sharing meta tags. ‘Hey mate, I’ve heard you can put some tags in and I’ll get the cool card when I share my pages on Twitter and Facebook?’ Uhhh sure, just chuck them in React Helmet and you’ll be sweet right? But hang on, it’s not working!

So what’s the deal, can you add social share meta tags with React Helmet?

Absolutely… with a caveat.

See, the problem is that the various social networks don’t run your JavaScript and so won’t see your dynamically updated meta tags as they’re added on page load.

So the answer is that you need a way to either statically generate, or server-side render your React app.

Whilst not exhaustive, here are a couple of ways you can do that.

React Snapshot

Probably the easiest if you’ve just got a Create React App (CRA) app, is to use React Snapshot.

React Snapshot takes your CRA app, follows any internal links it can find and generates a static site.

Just a simple npm install, update 3 lines and you’ve got a statically generated site.

static-site-generator-webpack-plugin

If you’ve got a custom Webpack setup, you can use the static-site-generator-webpack-plugin

This plugin formed the basis of Gatsby (mentioned next) before version 2.

Gatsby

As primarily a static site generator, Gatsby can do this out of the box. If you use the blog starter, you can use the SEO component

Even if you’re not using the blog starter, you can use the SEO component for inspiration and implement it in your site.

Next.js

Next.js has a feature called Static HTML Export which can statically render your site.

$ next build && next export

And if you’re not using getInitialProps, you might not even need to use next export.

Next.js has a feature called Automatic Static Optimization which will automatically prerender .html files for those it can.

Testing your social cards

Once you’ve set up your social card meta tags, you can test your meta tags using the following tools.

Is it ever OK to set state directly? A setState cheat sheet

Tue, 04 Feb 2020 08:30:00 GMT

One of the parts of React that trips people up is understanding state and how to update it. A lot of this confusion comes from the fact that setState is asynchronous.

As they write in the React docs:

Think of setState() as a request rather than an immediate command to update the component.

(Note: I’ll be referring to setState, but it applies equally to the useState hook which we’ll cover later)

So if I want to update immediately, I should just update the state object directly… right?

this.state.myValue = 'newValue';

NO! If you do this, you’re opting out of React’s handling, and your component won’t re-render.

The reason setState exists is to allow React to produce a more efficient and better experience for the end user. By batching the state updates, it can prevent needless re-renders and can allow React to do even more smart handling of different priority updates, aka concurrent mode.

OK, OK, I’ll use setState. But what about the weird updater function syntax? Surely I don’t need that?

Actually you do.

setState (and useState) have two forms, the updater function and the stateChange object syntax.

// Updater function
// pass a function to show what to update
this.setState(prevState => {
  return { count: prevState.count + 1 };
});

// stateChange object
// pass an object to show what to update
this.setState({ name: 'New Name' });

Which one should I use when?

Here’s a quick rule of thumb.

If your next state depends on the current state, use the updater function. Otherwise you’re fine to pass an object in the stateChange syntax.

For example:

updating a count (updater function)
updating an object (updater function)
toggling a boolean (updater function)
setting a new string value (stateChange object)

What about hooks and useState?

All the cases above apply equally to the useState hook.

Where you would use the updater function in setState, use the updater function in your hook setter function.

See the below examples in both setState and useState hook.

So how about that cheat sheet?

Most situations can be boiled down to one of a few examples, shown below:

Updating a count

Relies on previous state, use updater function

// this.setState()
this.setState(prevState => {
  return { count: prevState.count + 1 };
});

// useState hook
const [count, setCount] = useState(0);
setCount(prevCount => prevCount + 1);

Updating an object

Relies on previous state, use updater function

// this.setState()
this.setState(prevState => {
  return { config: { ...prevState.config, ...updatedValues } };
});

// useState hook
const [config, setConfig] = useState({});
setConfig(prevConfig => ({ ...prevConfig, ...updatedValues }));

Toggling a boolean

Relies on previous state, use updater function

// this.setState()
this.setState(prevState => {
  return { isEnabled: !prevState.isEnabled };
});

// useState hook
const [isEnabled, setIsEnabled] = useState(true);
setIsEnabled(prevIsEnabled => !prevIsEnabled);

Setting a new string value

Doesn’t rely on previous state, use stateChange object

// this.setState()
this.setState({ stringValue: "doesn't rely on previous state" });

// useState hook
const [stringValue, setStringValue] = useState('a default string');
setStringValue("doesn't rely on the previous state");

To wrap up

Component state is one of the main concepts you need to understand in React. Remember these simple rules and you’ll never be confused by setState again!

Never change state directly: always use setState or the useState setter function.
If your next state depends on the current state, use the updater function.

Avoid npm build errors by tracking your Node version with nvm

Tue, 21 Jan 2020 08:15:00 GMT

You know the feeling. You get a spare moment jump in and do some updates on an old app. But you try to get it running and the dreaded npm errors come up.

Oftentimes, the issue is that you’ve updated Node since you last ran it. Many issues with dependencies and npm can been easily solved by switching Node versions.

Easy, so just go back to the version of node you used when you originally developed it. But how do you know which version to jump to? Just guess and check?

Perhaps you can leave a note in your README saying “This works in Node version X.” And sure, that’s certainly one way of solving it.

But using nvm, we can track the version, as well as automate the switch back, and that’s what we’ll talk about today.

nvm is Node Version Manager. It primarily allows you to switch between isolated Node environments for different versions. If you’ve used rvm/rbenv in Ruby, it’s similar to that.

For our purposes today however, the killer feature is the .nvmrc file. .nvmrc is a file you commit with your project which contains a single line with the version of Node to use for this project.

In a project with an .nvmrc file, you can type nvm use and it will automatically switch back to the version you’ve stated. If you’re really keen, you can (and I’d say you should) also set up your shell to automatically nvm use when it encounters an .nvmrc file.

So start using nvm and .nvmrc in your projects and next time you’re returning to an old project, you’re just an nvm use away from being back up and running quick smart.

Arrow functions coming soon to PHP 7.4

Sun, 08 Sep 2019 18:00:00 GMT

The latest version of PHP 7.4 is coming soon. Release candidate 1 (the first of 6 planned RCs) was just released on 5th September and 7.4 is scheduled for General Availability on November 28, 2019.

If you’re like me, one of the cooler additions to look forward to coming to the language is arrow functions.

Arrow functions in ES6

If you’ve done any modern JavaScript, you’re probably familiar with arrow functions which came in ES6. In case you’re not, they look like this:

// in JavaScript arrow functions looks like this

// add numbers and return the result
const addSomeNumbers = (a, b) => a + b;

// echo out a message and return the input parameter
const doTheThing = theThing => {
  console.log(`I'm doing ${theThing}`);
  return theThing;
};

Arrow functions in PHP

prefix with fn
only single statement implicit return style arrow functions are implemented currently, no block based style like the first JavaScript example (although the block style is in the future scope - named multi-statement bodies in the RFC)
supports implicit variable binding, in other words, use is not required

Here are the examples from the RFC showing the arrow syntax and implicit variable binding:

// in PHP arrow functions look like this

// long version required currently
function array_values_from_keys($arr, $keys) {
    return array_map(
      function ($x) use ($arr) { return $arr[$x]; }, // long
      $keys
    );
};

// with arrow functions
function array_values_from_keys($arr, $keys) {
    return array_map(
      fn($x) => $arr[$x], // short
      $keys
    );
}

// note the difference
$long  = function ($x) use ($arr) { return $arr[$x]; };
$short = fn($x) => $arr[$x];

Try it out

I’ve created a simple test repo so you can try them out now.

Check out https://github.com/zorfling/php-74-arrow-functions

Build a GraphQL server with Laravel

Thu, 26 Jul 2018 21:24:32 GMT

One of the cool new things coming out from Facebook Open Source is GraphQL.

What is GraphQL?

GraphQL is a specification for communicating with APIs. You can think of GraphQL as an equivalent to REST, where GraphQL and REST are both just specifications for how to communicate with an API.

Why is GraphQL awesome?

GraphQL is awesome because it

has a single endpoint
allows for relational queries
eliminates over-requesting and under-requesting
can return all the data in one request
can mitigate the need for versioning as the client determines what it wants, not the server
can combine disparate data sources into a single API

How do we do it?

To use GraphQL, we need two parts. First we need a GraphQL server, and then we need a GraphQL client. In this post we’ll build out a GraphQL server in Laravel, and then in a following post, we’ll build a client app which uses our new server to display some data.

Let’s go!

We’ll be building upon Steve Luscher’s awesome zero-to-graphql talk and repository. In the talk he spun up a GraphQL endpoint in Django, Rails and Node inside 30 minutes.

It exposes a person model as follows:

class Person
{
    /** @var string */
    public $id;

    /** @var string */
    public $firstName;

    /** @var string */
    public $lastName;

    /** @var string */
    public $username;

    /** @var string */
    public $email;

    /** @var Person[] */
    public $friends;
}

It also provides two queries, allPeople to list all the people, and person to retrieve a particular Person by ID.

We’ll use Laravel and Laravel GraphQL to create our very own GraphQL server with Laravel that meets this schema.

Set up a new Laravel app

First up we’ll use the Laravel CLI to create a new application.

$ laravel new zero-graphql

Add a database

I’m using Laradock to serve and provide a MySQL database for our users, but you could just as easily use Homestead or a locally installed Mysql database with Valet.

Configuration of this step is beyond the scope of this post, but see the repo and Laradock docs for a configured Laradock setup, or the docs for Homestead or Valet

Add Laravel GraphQL

Next up we’ll add the Laravel GraphQL library to provide some GraphQL primitives, as well as the GraphiQL web based API client for testing the endpoint.

# Require the library
$ composer require folklore/graphql
# publish the graphql config and graphiql route
$ artisan vendor:publish --provider="Folklore\GraphQL\ServiceProvider"

Add Person Type

As GraphQL is a strongly typed definition, we need to define the types we’ll use, in our case the Person type. Thankfully the Laravel GraphQL library provides an artisan command to scaffold out our Person type.

$ ./artisan make:graphql:type PersonType

And then add our fields to the newly generated type file.

// app/GraphQL/Type/PersonType.php
...
return [
    'id' => [
        'type' => Type::nonNull(Type::int()),
        'description' => 'The id of the user'
    ],
    'first_name' => [
        'type' => Type::nonNull(Type::string()),
        'description' => 'The firstName of the user'
    ],
    'last_name' => [
        'type' => Type::nonNull(Type::string()),
        'description' => 'The lastName of the user'
    ],
    'username' => [
        'type' => Type::nonNull(Type::string()),
        'description' => 'The username of the user'
    ],
    'email' => [
        'type' => Type::nonNull(Type::string()),
        'description' => 'The email of user'
    ],
    'friends' => [
        'type' => Type::listOf(GraphQL::type('Person')),
        'description' => 'Person\'s friends'
    ]
];

Add People Data

So we have something for our GraphQL API to return, let’s add some people to the database. We’ll create our Person model and migrations and seeders for each of the People and Friendships tables. Again we can use artisan to scaffold out the files.

$ ./artisan make:model Person -m
$ ./artisan make:seeder PeopleTableSeeder
$ ./artisan make:seeder FriendshipsTableSeeder

We’ll add our columns to our people and our friendships table and some seed data for each.

// database/migrations/2018_04_16_045416_create_people_table.php
...
    public function up()
    {
        Schema::create('people', function (Blueprint $table) {
            $table->increments('id');
            $table->string('first_name');
            $table->string('last_name');
            $table->string('username');
            $table->string('email');
            $table->timestamps();
        });
    }
...

// database/seeds/PeopleTableSeeder.php

use Illuminate\Database\Seeder;

class PeopleTableSeeder extends Seeder
{
    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {
        DB::table('people')->insert([
            'id' => 1,
            'first_name' => 'Kyle',
            'last_name' => 'Reyes',
            'username' => 'kreyes',
            'email' => 'kyle.reyes@example.com',
        ]);
        DB::table('people')->insert([
            'id' => 2,
            'first_name' => 'Emma',
            'last_name' => 'Thomsen',
            'username' => 'ethomsen',
            'email' => 'emma.thomsen@example.com',
        ]);
        DB::table('people')->insert([
            'id' => 3,
            'first_name' => 'Crispim',
            'last_name' => 'Ramos',
            'username' => 'cramos',
            'email' => 'crispim.ramos@example.com',
        ]);
    }
}

// database/migrations/2018_04_16_051549_create_friendships_table.php
...
    public function up()
    {
        Schema::create('friendships', function (Blueprint $table) {
            $table->integer('person_id');
            $table->integer('friend_id');
            $table->timestamps();
        });
    }
...

// database/seeds/FriendshipsTableSeeder.php

use Illuminate\Database\Seeder;

class FriendshipsTableSeeder extends Seeder
{
    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {
        DB::table('friendships')->insert([
            'person_id' => 1,
            'friend_id' => 2,
        ]);
        DB::table('friendships')->insert([
            'person_id' => 1,
            'friend_id' => 3,
        ]);
        DB::table('friendships')->insert([
            'person_id' => 2,
            'friend_id' => 1,
        ]);
        DB::table('friendships')->insert([
            'person_id' => 2,
            'friend_id' => 3,
        ]);
        DB::table('friendships')->insert([
            'person_id' => 3,
            'friend_id' => 1,
        ]);
        DB::table('friendships')->insert([
            'person_id' => 3,
            'friend_id' => 2,
        ]);
    }
}

And finally our Person model

<?php
// app/Person.php

namespace App;

use Illuminate\Database\Eloquent\Model;

class Person extends Model
{
    public function friends()
    {
        return $this->belongsToMany('App\Person', 'friendships', 'person_id', 'friend_id');
    }
}

GraphQL Queries

Now we’re ready to add our GraphQL queries.

All People Query

The all people query allows us to list all people in the database.

# Query to return everyone
$ ./artisan make:graphql:query AllPeopleQuery

Update the name to allPeople

protected $attributes = [
    'name' => 'allPeople'
];

Update the type to a list of People

public function type()
{
    return Type::listOf(GraphQL::type('Person'));
}

And add our resolver

public function resolve($root, $args)
{
    return Person::all();
}

Person Query

The person query allows us to retrieve a single person from the database by id.

# Query to return a person by ID
$ ./artisan make:graphql:query PersonQuery

Update the name to person

protected $attributes = [
    'name' => 'person'
];

Update the type to a Person

public function type()
{
    return GraphQL::type('Person');
}

And add our resolver

public function args()
{
    // this allows us to query a particular id
    // it is passed to resolve() below
    return [
        'id' => ['name' => 'id', 'type' => Type::nonNull(Type::int())],
    ];
}

public function resolve($root, $args)
{
    if (isset($args['id'])) {
        return Person::where('id', $args['id'])->get()->first();
    }
}

GraphiQL

GraphiQL is a handy web based GraphQL client, baked in when we installed Laravel GraphQL. It’s similar to something like Postman which you might have used for testing REST APIs.

You can access it by going to http://localhost/graphiql.

Once you’re there, you can test out the queries we defined earlier.

allPeople Query

Get the username, first name and last name of all users, as well as their friends’ usernames.

Notice we can nest inside the query to return the username of each person’s friends as well. This is another of GraphQL’s benefits.

Request

{
  allPeople {
    username
    first_name
    last_name
    friends {
      username
    }
  }
}

Response

{
  "data": {
    "allPeople": [
      {
        "username": "kreyes",
        "first_name": "Kyle",
        "last_name": "Reyes",
        "friends": [
          {
            "username": "ethomsen"
          },
          {
            "username": "cramos"
          }
        ]
      },
      {
        "username": "ethomsen",
        "first_name": "Emma",
        "last_name": "Thomsen",
        "friends": [
          {
            "username": "kreyes"
          },
          {
            "username": "cramos"
          }
        ]
      },
      {
        "username": "cramos",
        "first_name": "Crispim",
        "last_name": "Ramos",
        "friends": [
          {
            "username": "kreyes"
          },
          {
            "username": "ethomsen"
          }
        ]
      }
    ]
  }
}

Person Query

Get the username of user id 1, along with the username, first name, last name and email of all his friends.

Request

{
  person(id: 1) {
    username
    friends {
      username
      first_name
      last_name
      email
    }
  }
}

Response

{
  "data": {
    "person": {
      "username": "kreyes",
      "friends": [
        {
          "username": "ethomsen",
          "first_name": "Emma",
          "last_name": "Thomsen",
          "email": "emma.thomsen@example.com"
        },
        {
          "username": "cramos",
          "first_name": "Crispim",
          "last_name": "Ramos",
          "email": "crispim.ramos@example.com"
        }
      ]
    }
  }
}

Other cool GraphQL endpoints

Here is a list of cool GraphQL endpoints to play around with and use for inspiration.

https://github.com/APIs-guru/graphql-apis

Next Steps

Check out the Github repo to clone the finished code.

And stay tuned for an upcoming post where we’ll build a client app which uses our new server to display the data!

Starting something new

Mon, 29 Jan 2018 10:26:00 GMT

EDIT (2020-09-19): I’ve folded the posts from Modern JS for PHP Devs back into chriscolborne.com here.

Over the last month or two, I’ve started something new. I’ve started a new blog over at https://www.modernjsforphpdevs.com.

I started the blog because after 6 wonderful years at The University of Queensland, I’ll be moving on to new things at the end of March.

My team has traditionally used quite a “safe” stack of Symfony and Backbone.js, so to learn all the modern tools, methods and libraries, and to ensure my skills are up to date, I need to do it on my own time. So what better way than to blog about it?

So who’s it for? I’m writing for someone like me. Maybe they’re a PHP dev who doesn’t get to use the coolest, newest stuff at work, but is still keen to try new things. They want to ensure their skills are up to date, and future proof their web development career.

It’s early days yet, but I’ve already got a couple posts about using webpack with Symfony (via Webpack Encore) to build and use React components within Twig, and a “reader question” about why it might be time to hang up the dollar sign and ditch jQuery.

I’ll probably cross post back here occasionally, but I’d love it if you drop over to Modern JS for PHP Devs and have a read!

Why React or Vue and not jQuery?

Sun, 28 Jan 2018 22:55:32 GMT

One of my colleagues has been playing devil’s advocate and hitting me with variations of the following question for a week or two now.

His question, “Chris, jQuery is so quick and easy to get going, why would I possibly use something like React or Vue that has such a steep setup curve? And it has so many plugins! And why wouldn’t I just keep using jQuery anyway? There’s nothing wrong with it!”

So there’s a couple of questions in that, so I’ll try here to break it down. I’ll start with the latter.

Why shouldn’t I keep using jQuery?

So the bottom line here is, because you probably don’t need it anymore. jQuery was born in an age of ES5, when it was truly difficult to maintain browser compatibility, when functional paradigms were the bastion of mad scientist types (I’m looking at you, Haskell devs!) and as JavaScript as a web language was growing from the dark old days of DHTML, where JavaScript was used for nothing more than image rollovers, to a burgeoning new age of web application.

If you really don’t believe that you don’t need it, check out the great list at You Might Not Need jQuery, and while you’re at it, reconsider underscore/lodash with You Don’t Need Lodash/Underscore and the accompanying eslint plugin.

jQuery is so quick, how can React or Vue compete, and why bother?

The other argument for jQuery is speed to implement. We’ve all seen How it feels to learn JavaScript in 2016, and yes, the more modern libraries need a build step. And yes, if you just want to make a button click hide a div, one line of jQuery can get that out quick smart, no build required.

But with the advent of zero-config (#0CJS) toolkits such as Create React App and vue-cli, a fully featured build process and install is one node command away, then you’re off to the races, with added bells and whistles such as hot module reload and the chance to use modern JavaScript features from ES6 and beyond.

And with tools such as CodeSandbox you can hack out a quick prototype quicker than you can say <script src="https://…

But really, it’s what comes next that is the reason to leave jQuery for React or Vue. How often have you only made a single button click hide a div?

Does this look familiar?

Image by Manu Cornet

The reason to use one of these new component based libraries is that it makes you design your app in a different way (or actually design it at all). You think in components, you have a defined structure and data model, with predictable and maintainable structure.

But what about all the plugins? Won’t someone please think of the plugins?!

Yes, jQuery has been around so long it’s basically got the “there’s ~~an app~~ a plugin for that” covered.

But both Vue and React communities are growing so fast, and their component based systems are tailor made for plug and play functionality.

Next time you’re looking for a drop in feature, try these awesome lists for React and Vue.

Still not enough? You could always build a wrapper component for your fave library. Check out this post over on Medium to find out How to Use jQuery Libraries in the React Ecosystem

So this is the end?

jQuery is an amazing library that changed the JavaScript landscape. But maybe it’s time to hang up the dollar sign and use some pure JavaScript. Add a component based library and you’ll never think about your code the same way again.

React starter repo with Symfony 4 and Webpack Encore

Sun, 14 Jan 2018 23:17:32 GMT

In the last post, we introduced Webpack Encore and why it makes for a quick start for modern JS with Symfony.

In this post, we’ll set up a basic starter repo for a React frontend using Symfony and Webpack Encore.

As a bonus, as Symfony 4 was released at the end of last year, let’s use the latest and greatest and start with that.

Set up a new Symfony 4 project

I’ll assume that you’ve got a basic understanding of composer and Symfony, but if not, go check out the Symfony quick start.

Ready? Ok let’s go!

$ composer create-project symfony/skeleton my-project

Tip: you’ll need composer to be installed globally for this to work.

This will create a directory called my-project in the current directory with a Symfony skeleton site, and use composer to install all the dependencies.

Next we’ll set up some basic dependencies for our site.

$ composer require annotations twig asset
$ composer require server --dev

Configure Webpack Encore to add React

Webpack Encore is simply a node package, so we can install it using npm (or yarn). But since we’re using composer flex, let’s use it to bootstrap things.

# Create package.json, webpack.config.js and add node_modules to .gitignore
$ composer require encore

# Add react deps
$ yarn add --dev react react-dom prop-types babel-preset-react

# Actually install node package
$ yarn install

And finally, we need to enable the React preset in our webpack.config.js.

// webpack.config.js
…
Encore
  …
  .enableReactPreset();

Set up a demo JSON API

We’ll set up a basic JSON endpoint to display some dummy data. We’ll hard code it for this demo, but you could hit the database, or other APIs at this point.

// src/Controller/DefaultController.php
/**
 * @Route('/data')
 */
public function dataAction()
{
    return new JsonResponse([
        [
            'id' => 1,
            'author' => 'Chris Colborne',
            'avatarUrl' => 'http://1.gravatar.com/avatar/13dbc56733c2cc66fbc698cdb07fec12',
            'title' => 'Bitter Predation',
            'description' => 'Thirteen thin, round towers …',
        ],
        [
            'id' => 2,
            'author' => 'Louanne Perez',
            'avatarUrl' => 'https://randomuser.me/api/portraits/thumb/women/18.jpg',
            'title' => 'Strangers of the Ambitious',
            'description' => "A huge gate with thick metal doors …",
        ],
        [
            'id' => 3,
            'author' => 'Theodorus Dietvorst',
            'avatarUrl' => 'https://randomuser.me/api/portraits/thumb/men/49.jpg',
            'title' => 'Outsiders of the Mysterious',
            'description' => "Plain fields of a type of grass cover …",
        ],
    ]);
}

Set up a React frontend which uses it

Ok, finally we need to setup React to use our JSON endpoint.

First add a div with an id of root to index.html.twig. This will be our react mount point.

<!-- templates/Default/index.html.twig -->
…
<div id="root"></div>

Next we’ll create our react component, and fetch the data from our endpoint. We’ll use Material UI for quick good looking components, but you could just as easily build your own.

We’ll include Material UI with yarn.

yarn add material-ui

And we’ll create an ItemCard component using Material UI …

// assets/js/Components/ItemCard.js
import React from 'react';
import { Card, CardHeader, CardTitle, CardText } from 'material-ui/Card';

const ItemCard = ({ author, avatarUrl, title, subtitle, style, children }) => (
  <Card style={style}>
    <CardHeader title={author} avatar={avatarUrl} />
    <CardTitle title={title} subtitle={subtitle} />
    <CardText>{children}</CardText>
  </Card>
);

export default ItemCard;

… and an App component which lists a few cards based on our endpoint above.

// assets/js/app.js
import React from 'react';
import ReactDOM from 'react-dom';
import MuiThemeProvider from 'material-ui/styles/MuiThemeProvider';

import ItemCard from './Components/ItemCard';

class App extends React.Component {
  constructor() {
    super();

    this.state = {
      entries: [],
    };
  }

  componentDidMount() {
    fetch('/data')
      .then(response => response.json())
      .then(entries => {
        this.setState({
          entries,
        });
      });
  }

  render() {
    return (
      <MuiThemeProvider>
        <div style={{ display: 'flex' }}>
          {this.state.entries.map(
            ({ id, author, avatarUrl, title, description }) => (
              <ItemCard
                key={id}
                author={author}
                title={title}
                avatarUrl={avatarUrl}
                style={{ flex: 1, margin: 10 }}
              >
                {description}
              </ItemCard>
            )
          )}
        </div>
      </MuiThemeProvider>
    );
  }
}

ReactDOM.render(<App />, document.getElementById('root'));

Finally we need to add our application JavaScript as an entrypoint to our webpack.config.js.

// webpack.config.js
  …
  .addEntry('js/app', './assets/js/app.js')

We’ll then include the built JavaScript in our twig template and we’re done!

<!-- templates/Default/index.html.twig -->
{% extends 'base.html.twig' %}

{% block body %}
<div id="root"></div>
<script type="text/javascript" src="{{ asset('build/js/app.js') }}"></script>
{% endblock %}

To view the final product, we need to compile the assets using Webpack Encore.

$ yarn run encore dev
# yarn run encore dev --watch to automatically rebuild on every change

And then use the Symfony WebServerBundle we installed above to start the server.

# This will start a server at http://127.0.0.1:8000.

$ ./bin/console server:start # stop with server:stop

The final result is three React cards that fill the available space on the page.

Check out the Github repo to clone the finished starter repo.

Modern JS in Symfony with Webpack Encore

Mon, 27 Nov 2017 22:22:32 GMT

Symfony 2 introduced Assetic to provide asset management. Back in 2011, in the days of server rendered application, this pure PHP solution was good choice, but in 2017, Single Page Apps (SPA) and other more heavily javascript driven websites are the norm.

Assetic didn’t quite handle this modern world of JavaScript, where WebPack is the tool du jour for packaging and transpiling javascript, and including static JS straight from the file system is becoming somewhat of a faux pas.

That’s why, in June this year, Symfony released Webpack Encore.

Webpack Encore is an abstraction over webpack which simplifies the configuration and provides a number of common transformations for a modern frontend toolchain. It brings best of breed asset bundling to your Symfony applications.

It also works great with Symfony and Twig, with the JSON manifest strategy introduced in Symfony 3.3. With one line of code, you can introduce asset versioning, and even configure a CDN.

Here’s an example webpack Encore webpack.config.js and the webpack config it generates. Notice the Webpack Encore is 23 lines (including comments) vs 168 lines of pure webpack config. 😮

// Webpack Encore - webpack.config.js
var Encore = require('@symfony/webpack-encore');

Encore
  // the project directory where all compiled assets will be stored
  .setOutputPath('web/build/')
  // the public path used by the web server to access the previous directory
  .setPublicPath('/build')
  // will create web/build/app.js and web/build/app.css
  .addEntry('app', './assets/js/app.js')
  // allow sass/scss files to be processed
  .enableSassLoader()
  // allow legacy applications to use $/jQuery as a global variable
  .autoProvidejQuery()
  .enableSourceMaps(!Encore.isProduction())
  // empty the outputPath dir before each build
  .cleanupOutputBeforeBuild()
  // show OS notifications when builds finish/fail
  .enableBuildNotifications()
  // create hashed filenames (e.g. app.abc123.css)
  .enableVersioning();

// export the final configuration
module.exports = Encore.getWebpackConfig();

// Pure webpack - webpack.config.js
module.exports = {
  context: '/Users/modern-js/Code/php/encore-test',
  entry: { app: './assets/js/app.js' },
  output: {
    path: '/Users/modern-js/Code/php/encore-test/web/build',
    filename: '[name].[chunkhash].js',
    publicPath: '/build/',
    pathinfo: false,
  },
  module: {
    rules: [
      {
        test: {},
        exclude: {},
        use: [
          {
            loader: 'babel-loader',
            options: {
              cacheDirectory: true,
              presets: [
                [
                  'env',
                  {
                    modules: false,
                    targets: { browsers: '> 1%', uglify: true },
                    useBuiltIns: true,
                  },
                ],
              ],
              plugins: [],
            },
          },
        ],
      },
      {
        test: {},
        use: [
          {
            loader:
              '/Users/modern-js/Code/php/encore-test/node_modules/extract-text-webpack-plugin/dist/loader.js',
            options: { omit: 1, remove: true },
          },
          { loader: 'style-loader' },
          {
            loader: 'css-loader',
            options: { minimize: true, sourceMap: false, importLoaders: 0 },
          },
        ],
      },
      {
        test: {},
        loader: 'file-loader',
        options: {
          name: 'images/[name].[hash:8].[ext]',
          publicPath: '/build/',
        },
      },
      {
        test: {},
        loader: 'file-loader',
        options: { name: 'fonts/[name].[hash:8].[ext]', publicPath: '/build/' },
      },
      {
        test: {},
        use: [
          {
            loader:
              '/Users/modern-js/Code/php/encore-test/node_modules/extract-text-webpack-plugin/dist/loader.js',
            options: { omit: 1, remove: true },
          },
          { loader: 'style-loader' },
          {
            loader: 'css-loader',
            options: { minimize: true, sourceMap: false, importLoaders: 0 },
          },
          { loader: 'resolve-url-loader', options: { sourceMap: false } },
          { loader: 'sass-loader', options: { sourceMap: true } },
        ],
      },
    ],
  },
  plugins: [
    {
      filename: '[name].[contenthash].css',
      id: 1,
      options: { allChunks: false },
    },
    { entriesToDelete: [] },
    {
      opts: {
        basePath: 'build/',
        publicPath: '/build/',
        fileName: 'manifest.json',
        stripSrc: null,
        transformExtensions: {},
        writeToFileEmit: true,
        cache: null,
      },
    },
    {
      options: {
        debug: false,
        options: {
          context: '/Users/modern-js/Code/php/encore-test',
          output: { path: '/Users/modern-js/Code/php/encore-test/web/build' },
        },
        test: {},
      },
    },
    {
      options: {
        hashFunction: 'md5',
        hashDigest: 'base64',
        hashDigestLength: 4,
      },
    },
    { algorithm: 'md5', digest: 'hex' },
    {
      definitions: { $: 'jquery', jQuery: 'jquery', 'window.jQuery': 'jquery' },
    },
    {
      paths: ['**/*'],
      options: {
        root: '/Users/modern-js/Code/php/encore-test/web/build',
        verbose: false,
        allowExternal: false,
        dry: false,
      },
    },
    { definitions: { 'process.env': { NODE_ENV: '"production"' } } },
    { options: { sourceMap: false } },
    {
      options: { title: 'Webpack Encore' },
      lastBuildSucceeded: false,
      isFirstBuild: true,
    },
    {
      compilationSuccessInfo: { messages: [] },
      shouldClearConsole: false,
      formatters: [null, null, null, null, null, null],
      transformers: [null, null, null, null, null, null],
    },
    {
      outputPath: 'web/build',
      friendlyErrorsPlugin: {
        compilationSuccessInfo: { messages: [] },
        shouldClearConsole: false,
        formatters: [null, null, null, null, null, null],
        transformers: [null, null, null, null, null, null],
      },
    },
  ],
  performance: { hints: false },
  stats: {
    hash: false,
    version: false,
    timings: false,
    assets: false,
    chunks: false,
    maxModules: 0,
    modules: false,
    reasons: false,
    children: false,
    source: false,
    errors: false,
    errorDetails: false,
    warnings: false,
    publicPath: false,
  },
  resolve: { extensions: ['.js', '.jsx', '.vue', '.ts', '.tsx'], alias: {} },
};

Stay tuned for an upcoming post where I’ll set up a simple starter repo for Symfony with a React frontend.

Git and Mercurial Cheat Sheet

Sun, 01 Feb 2015 12:23:44 GMT

Our team supports a large number of applications, ranging from legacy apps in SVN, to more recent apps in Mercurial, to our latest projects in Git.

Inspired by the excellent (but very thorough) Mercurial to Git Rosetta stone, here is a bare bones, back to basics, KISS mapping between Hg and Git of the most common operations I use on a day to day basis.

Init

hg init git init

Add Remote

edit paths in .hg/hgrc git remote add origin git@github.com:<username>/<repo-name>.git

Clone

hg clone <url> git clone <url>

Pull

hg pull -u git pull

Push

hg push git push

Diff

`hg diff`	`git diff` OR `git diff HEAD` (to diff both staged and unstaged)
`hg diff -r 20:21 ./src/filename`	`git diff <hash>..<hash> -- ./src/filename`
`... coming ...`	`git diff master...<hash>` (to diff RHS with common ancestor)

See incoming/outgoing commits

`hg in`	`git fetch && git log master..FETCH_HEAD`
`hg out`	`git fetch && git log FETCH_HEAD..master`

Commit

hg ci -m"Commit message" git commit -am"Commit message"

Interactively commit part of working copy

hg record git add --patch && git commit

Branches

`hg branch <branch-name>`	`git checkout -b <branch-name>`
`hg up <branch-name>`	`git checkout <branch-name>`
`hg push --new-branch`	`git push origin <branch-name> -u`
`hg branches`	`git branch -a`
`hg commit --close-branch`	# delete local branch `git branch -d <branch-name>` then # delete remote branch (NB the colon) `git push origin :<branch-name>` also # remove remote tracking branches that no longer exist `git fetch -p`
# show commits from branch `hg log -b <branch-name>`	`git reflog show <branch-name>`

Merge

hg merge <branch-name or commit number/hash> git merge <branch-name or commit hash>

Shelve/stash

`hg shelve`	`git stash`
`hg unshelve`	`git stash pop`

Clear working copy

hg up -C git checkout -f

Rollback

hg rollback git reset HEAD~

Remove ignored files that are already committed

hg forget "set:hgignore() and not ignored()" git ls-files --ignored --exclude-standard | xargs git rm --cached

Markdown to Confluence with Ruby and SOAP

Sun, 18 Jan 2015 12:23:44 GMT

This post follows on from my previous post about the SharePoint to Markdown ripper. You can go back and read it if you’d like.

So once I’d extracted our SharePoint wiki as markdown, it was time to find a way to import that into Confluence.

I’d used the markdown editor of the Insert Markup function which worked great, but for ~300 separate pages, I needed a slightly more efficient method than copy paste into a text box.

My first thought was to try find a SOAP or REST API to perform the page creation. I presumed since the Insert Markup function could do it, there might be an API for it. However I wasn’t able to find anything to provide Markdown page creation. So I needed to find another answer.

I found a GitHub repo for a markdown to confluence uploader designed to import a single markdown file into an existing page using the Confluence SOAP api.

Obviously I needed something to iterate through a directory of files, and to create, rather than use an existing page, but it seemed like a good start.

So I forked the repo and added an option to provide a parent page to house the newly created pages.

I then used awk and xargs to iterate through the files importing under a page I’d created for the legacy wiki.

ls output/ | awk '{ print "output/" $1 }' | xargs -I {} ./md2confl.rb -s SS -c http://myconfluence.net -t1671258 --markdownFile {}

This imported my 300 legacy SharePoint pages into Confluence in a flash.

However, there were some conversion issues, so this is still a work in progress.

The script I forked uses the markdown2confluence gem, which by it’s own admission is not a 100% conversion.

In fact there is an outstanding issue regarding the nested list problem that I faced on the gem’s GitHub page. I intend to fork and submit a pull request to solve this issue when I get a chance.

All in all, this gets me almost to the point of being able to move across to Confluence. I hope it might help someone else in the same boat.

Get the source

Fork or clone the markdown to confluence uploader on GitHub. Issues and pull requests welcome!

Sharepoint Scraping with Python, Mechanize, and Beautiful Soup

Sun, 11 Jan 2015 00:00:00 GMT

Over the Christmas break, I decided to write a SharePoint scraper in Python.

At work we have a SharePoint based wiki that is universally disliked and thus is poorly used despite containing a lot of useful information. I’ve been trialling Atlassian Confluence as a replacement and it seems a much better solution. However we don’t want to run multiple wikis so I’ll need to copy the existing data across.

Gaining permission to use any APIs that SharePoint might expose over the Christmas break was obviously not going to happen, so I decided to use Python and Mechanize to scrape the wiki.

Problem 1 - Authentication

After some quick research, I found our SharePoint instance doesn’t accept basic authentication, but rather only uses the proprietary NTLM protocol.

I found a Python module, python-ntlm, which provided an authentication handler for NTLM.

Following some discussion on Stack Overflow here, I was able to hook the NTLM handler up to the Mechanize module.

import mechanize
from ntlm import HTTPNtlmAuthHandler

passman = mechanize.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, user, password)
# create the NTLM authentication handler
auth_NTLM = HTTPNtlmAuthHandler.HTTPNtlmAuthHandler(passman)

# create and install the opener
browser = mechanize.Browser()
browser.add_handler(auth_NTLM)
browser.open(url)

I ran into a couple of hurdles with this approach.

Firstly, Mechanize correctly follows the robots.txt. However, since this is a once off scrape off our own site, I was happy to bypass the robots.txt.

browser.set_handle_robots(False)

Secondly, Mechanize seems to have a bug when using NTLM authentication. With a patch described here, I was able to get Mechanize up and running. A pull request is open on GitHub, however it hasn’t been merged.

Problem 2 - Traversing and filtering the tree

Since SharePoint HTML is, well … not pretty, not to mention there’s a bunch of site chrome we don’t want to scrape, I needed to do some cleansing.

I used the module Beautiful Soup to filter the tree and clean up the code.

Beautiful Soup’s tagline reads as follows:

You didn’t write that awful page. You’re just trying to get some data out of it. Beautiful Soup is here to help. And that certainly fit my use case.

I filtered to within a particular div (the obviously named ctl00_PlaceHolderMain_WikiField for those of you playing at home) that excluded all the extraneous SharePoint navigation stuff.

Beautiful Soup can use a number of different parsers, depending on your usage.

I started off using the default Python html.parser, however the SharePoint code was too dirty for it, and it fell down on a number of pages. So I changed to the much more permissive html5lib and found all of my pages scraped correctly.

Once I’d filtered and scraped my pages, I had more or less well formed HTML of the page content only, which led to my final challenge.

I then recursively iterated through each of the links parsing local SharePoint links.

Problem 3 - Converting to Markdown

I decided early on that converting to Markdown would probably be a good idea. This would allow me to strip it back to basics, getting rid of the nasty font tags, inline styles, and other junk that had polluted the SharePoint pages.

This proved to be a pretty easy task. I used the html2markdown module.

I ran my newly well formed HTML through it and voila, Markdown!

Get the source

Fork or clone the sharepoint-ripper on GitHub. Issues and pull requests welcome!

Stay tuned for my next post where I import the exported Markdown into Confluence.

Hello World!

Fri, 02 Jan 2015 08:01:37 GMT

Hi everyone, I’m Chris. I’m a Software Engineer in Brisbane, Australia.

As 2015 begins, I’ve decided to start a blog showcasing some of my work and hobbies here at ChrisColborne.com.

I’ll post a bunch of stuff, but mainly tech.

chriscolborne.com

Don't use method syntax for your TypeScript function types!

Typing Functions

1. Function type expression syntax

2. Call signature syntax

3. Method syntax

strictFunctionTypes flag in TSConfig

What should you use?

How can I run some code after changing state with React hooks?

Method 1 - useEffect hook

Method 2 - useReducer hook

Object Destructuring 101

aka what on Earth does doIt({ x: x2 = getX() } = {}) mean?!

Level 1: Basic Destructuring

Level 2: Renaming Destructuring

Level 3: Multi-Level Destructuring

Level 4: Destructuring Defaults

Full Circle

Which Headless CMS should I use for a new Gatsby site?

What is Gatsby?

What is the JAM Stack?

What is a headless CMS?

What are the options?

Open source options

Software as a Service (SaaS)

Which one should you use?

Can I use React Helmet to add social card meta tags for Twitter and Facebook?

React Snapshot

static-site-generator-webpack-plugin

Gatsby

Next.js

Testing your social cards

Is it ever OK to set state directly? A setState cheat sheet

So if I want to update immediately, I should just update the state object directly… right?

OK, OK, I’ll use setState. But what about the weird updater function syntax? Surely I don’t need that?

Which one should I use when?

What about hooks and useState?

So how about that cheat sheet?

Updating a count

Updating an object

Toggling a boolean

Setting a new string value

To wrap up

Avoid npm build errors by tracking your Node version with nvm

Arrow functions coming soon to PHP 7.4

Arrow functions in ES6

Arrow functions in PHP

Try it out

Build a GraphQL server with Laravel

What is GraphQL?

Why is GraphQL awesome?

How do we do it?

Let’s go!

Set up a new Laravel app

Add a database

Add Laravel GraphQL

Add Person Type

Add People Data

GraphQL Queries

All People Query

Person Query

GraphiQL

allPeople Query

Request

Response

Person Query

Request

Response

Other cool GraphQL endpoints

Next Steps

Starting something new

Why React or Vue and not jQuery?

Why shouldn’t I keep using jQuery?

jQuery is so quick, how can React or Vue compete, and why bother?

But what about all the plugins? Won’t someone please think of the plugins?!

So this is the end?

React starter repo with Symfony 4 and Webpack Encore

Set up a new Symfony 4 project

Configure Webpack Encore to add React

Set up a demo JSON API

aka what on Earth does `doIt({ x: x2 = getX() } = {})` mean?!