Progressive Web Apps

5.2 TMDB API, Fetch, and Security

The Movie DB Review

TMBD API (opens new window) was something that you used last semester. It is the API that can be used to find information on TV Shows and Movies through fetch calls.

We are using version 3 of the API. It requires that you register for a key. You should already have a key from the first semester. You can log in to the site or use the forgot password feature to reset your password and login. Then you will be able to go to your account page (opens new window) and get your key.

There are many different endpoints that you can use to search for different things.

All API calls begin with the url https://api.themoviedb.org/3 and then add the different end points.

They all need parameters to be passed through the query string as part of the URL too. Your key needs to be used on every call, like this ?api_key=thiswouldbeyourapikey.

As an example, if you wanted to search for an actor, you would do a fetch call to the /search/person end point, as detailed in the documentation for the /search/person endpoint (opens new window)

const BASE = `https://api.themoviedb.org/3`;
const searchPeople = `/search/person`;
const APIKEY = `thiswouldbeyourapikey`;

//do a search for actors whose name includes `emilia`
let name = 'emilia';
let url = BASE + searchPeople + `?api_key=${APIKEY}&query=${name}`;

fetch(url)
  .then((resp) => {
    if (!resp.ok) throw new Error(resp.statusText);
    return resp.json();
  })
  .then((content) => {
    //content is your javascript object back from the api
    //there will be a property called results that will hold an array of matches
    content.results.forEach((actor) => {
      console.log(actor); //output each matching actor Object in the console
    });
  })
  .catch((err) => {
    //display something about the error for the user
  });

Depending on the API call that you make, you will get different response objects with different properties. However, nearly all of them have a top level property called results that will be an array of matches to your query.

Handling Images

Movies, TV shows, and actors will nearly all have images that can be displayed. Often there will be multiple sizes of images. Image reference (opens new window).

The base URL for all images is https://image.tmdb.org/t/p. After the base value there will be a folder that indicates the image size. It will be something like /w500 for images that are 500px wide. The final part will be a property that includes path as part of the name. poster_path or background_path are common property names for the image. Different types of image paths have different sizes available. Reference the list below to see the options.

In the end you will end up with a URL like https://image.tmdb.org/t/p/w500/kqjL17yufvn9OVLyXYpvtyrFfak.jpg.

To get the list of sizes and the different possible URL parts for images you can make a call to the configuration endpoint - https://api.themoviedb.org/3/configuration?api_key=<<api_key>>. Here is what it will return.

{
  "images": {
    "base_url": "http://image.tmdb.org/t/p/",
    "secure_base_url": "https://image.tmdb.org/t/p/",
    "backdrop_sizes": ["w300", "w780", "w1280", "original"],
    "logo_sizes": ["w45", "w92", "w154", "w185", "w300", "w500", "original"],
    "poster_sizes": ["w92", "w154", "w185", "w342", "w500", "w780", "original"],
    "profile_sizes": ["w45", "w185", "h632", "original"],
    "still_sizes": ["w92", "w185", "w300", "original"]
  }
}

Need to know more about template strings?

Need to know more about Error objects and throwing Errors?

Fetch API Beyond the Basics

When you make fetch calls to the server it becomes important to understand exactly what you are doing and how all the parts fit together. The aim of this section is to give you a wider perspective about how all the parts fit together.

Originally API calls to the server were accomplished with a thing called XMLHttpRequest that was introduced in Internet Explorer 5.5. It was a way to fetch XML documents from the server. Which lead to the coining of the term AJAX - Asynchronous JavaScript And Xml in 2005, to describe this approach to client side web development.

It was the driving technology behind the rise of Google Maps. Combined with the ability to drag elements on the screen, google maps could load new locations and get new map tiles without every having to reload the whole page. MapQuest was the first big online mapping site. They had 8 buttons around the map and you had to click the button for the direction you wanted to move and then wait for the whole page to reload.

XMLHttpRequest is still supported in all browsers and is part of some libraries. It's good to be familiar with it. Here is a video about how to use it to make AJAX calls

fetch was developed as an enhanced and more secure way to pass things between the client and the server. It was based on Promises. The Fetch API included access to all the component parts of the request and response.

HTTP is a protocol used by clients and servers to talk to each other. It defines a way to bundle files and transfer them between computers.

It is made up of Requests and Responses.

Every HTTP Request and HTTP Response has a Head, a Body, and a Footer. We never see the footer. It is used to verify that things are reassembled properly and that no data is missing.

The Head holds the meta information about the request or response. What address is it being sent from, what address it is being sent to, whether it is encrypted, what type of file is contained in the body, the file size of the body, the encoding type, if it is compressed, cookies, what is the expiry date of the response, and much more. All the values in the Head are called Headers.

The QueryString is one of the Headers. It is a string that contains a series of name value pairs. There is an = sign between each name and value. And there is an & ampersand between each of the pairs. The values in the QueryString need to be URL encoded to avoid causing issues with other programs, through special characters, who might read the string.

The official object for a QueryString is a URLSearchParams (opens new window) object.

Cookies that we covered in week 2, are also one of the header values.

The Method verb (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS) is one of the Headers. The Method is important on the Server because it gives the server an idea of how to handle the request. You will learn more about this in MAD9124.

The Body is the container for the file that is being uploaded or downloaded. The Body can be empty if there is no file or data being uploaded or downloaded. With a GET Request the Body is always empty.

The two places that you can put information to send with a Request or Response are in the QueryString or in the Body. The QueryString is limited to about 2000 characters. The Body is limited in total size on most servers to files that are roughly 20MB. This limit can be changed through server-side settings.

Request Objects

When you make a fetch call, very often you are only providing a URL to the method. However, the fetch method will actually create a new Request() object on your behalf, using all the default values plus your URL.

If you need to you can create your own Request object.

let request = new Request();
//fetch also accepts a Request object instead of a URL object or URL string.
fetch(request)
  .then((response) => {})
  .then((body) => {})
  .catch(console.warn);

Inside the Request object you can define what the headers are and what the body is.

MDN reference for Request Object (opens new window). You can use this reference to find all the properties and methods of a Request object.

Response Objects

Typically you will be working with a response object that gets returned to the first then method after a fetch.

If you need to, like in a Service Worker when you want to send something to the browser that you are creating in response to a request, you can create your own Response Object.

let response = new Response();

MDN reference for a Response Object (opens new window). You can use this reference page to find all the properties and methods for a Response Object.

Body Objects

The Body object is the container for any file or large block of data being transferred between a client and a server. Both the Response and the Request objects have a body property that is used to access the Body object.

MDN reference for the body property of the Response object (opens new window)

The body property can contain one of the following data types:

• Blob (opens new window) - for binary files like images
• BufferSource (opens new window) - like an array of data but binary
• FormData (opens new window) - generally used for the information from a form
• ReadableStream (opens new window) - large chunk of text or numerical data
• URLSearchParams (opens new window) - Query String formatted string
• USVString (opens new window) - an encoded string

json(), text(), and blob() Methods

When a Response comes back to the browser, to either your main script or a service worker, the most common datatypes that we receive are:

• a json file
• a text file (CSS, XML, or HTML)
• an image (Blob)

Because of that, there are three specific methods that we can use to extract the contents of those files, from the body of the Response. We use the Response.json() method to convert the contents of the JSON file into a JavaScript Object. We use the Response.text() method to read the contents of the CSS, XML, or HTML file into a string. We use the Response.blob() method to extract the binary data from a binary file (like an image) into a Blob object.

All three of the methods are asynchronous and return a Promise. So, we use them inside a then() method and return the Promise that they create. That way it gets passed to the next then() in the chain. The next then() will receive the Object, String, or Binary content from the method.

fetch(request)
  .then((response) => {
    //only the first return actually runs
    return response.text();
    return response.blob();
    return response.json();
  })
  .then((body) => {
    //body is the formatted contents returned from one of those methods
  });

If we want to use the Blob as the source for an image element on our page then we need to use the URL.createObjectURL() method.

document.getElementById('dynamicImage').src = URL.createObjectURL(blob);

Header Objects

The Header object is like an array of name value pairs. Many of the headers will be created by the browser and sent automatically. These are values that you are not allowed to edit. Things like the ip address cannot be altered in your script.

List of restricted header names (opens new window)

In addition, you are not allowed to programmatically edit the set-cookie or set-cookie2 headers in a Response object.

The Header object has an append method that we can use to add or edit header values.

let h = new Headers();
h.append('content-type', 'application/json');
h.append('content-length', 13234); //file size in bytes

When we are uploading a file, it means that we are adding a file to the Body and we should include headers that indicate the type and size of the attached file.

Here is a list of possible headers (opens new window)

Many of the existing headers can also be accessed through the .headers property on the Request and Response objects and its .get() method.

fetch(url).then((response) => {
  response.headers.get('Content-Type');
  response.headers.get('Content-Encoding');
  response.headers.get('Age');
  response.headers.get('Date');
  response.headers.get('X-my-custom-header');
});

The other methods on the Header object include has(), entries(), and delete(). Here is the link to the get method (opens new window).

It is worth noting that not all headers are accessible through JavaScript. This is for security reasons. Some are not able to be changed or deleted through script and some are not allow to be read.

URL and Location Objects

URLs can be thought of as just strings but they do have clearly defined parts: the domain, the path, the hash, the querystring(search), the protocol, and the port.

JavaScript has a top level object window.location that contains all these parts. If you console.log the location object you will see all the different components that make it up. One way that you could built a url is to take or create a Location object and set the values for its different components and then use the full value of the href property string.

MDN reference for Location Object (opens new window)

Another approach is to use the URL object to create one. It takes two parameters that make it work a lot more like a typical development situation, where you have a base url and then an endpoint with or without a QueryString.

const BASEURL = 'http://somedomain.com';
//a global variable that holds the base url for your API

let endpoint = '/api/people/23?api-key=8768374823';
let url = new URL(endpoint, BASEURL);
//first we have a url object to pass to a Request or fetch call.

let str = url.toString();
//here we have a String built from the url object

let req = new Request(url); // or new Request(str)
//here we have a Request object with the url inside it

fetch(url).then((resp) => {
  //we can pass a url directly to fetch
  console.log('fetch with url', resp.status);
});

fetch(req).then((resp) => {
  //we can pass the request object that was given a url or string
  console.log('fetch with url in request', resp.status);
});

fetch(str).then((resp) => {
  //we can pass the string to the fetch
  console.log('fetch with url as string', resp.status);
});

The URL property of a Request or Location object, just like the URL object itself has a bunch of properties that you can access to directly read the different parts of a URL. (Otherwise you would have to parse the string yourself.)

• hash is the part that begins with #.
• host is a hostname with a port (if specified).
• hostname is the host without the port.
• href is the entire url string with all the parts.
• origin includes the scheme, domain, and port.
• pathname is the path and filename starting with /.
• port is the port number part of the url. Eg: 80 or 5500.
• protocol is the scheme, like http:, ftp:, https:, blob:, or file:.
• search is the querystring for the url and is either an empty string or includes the ? at the start.
• searchParams is a read-only SearchParams object with the elements in the querystring.

MDN reference for URL (opens new window)

FormData Objects

The FormData object is a great way to bundle an entire HTML form, including any hidden input elements, into a format that can be set as the value for a Request body.

let myform = document.getElementById('myform'); //a form element on the page
let fd = new FormData(myform);
//this will append all the name value pairs for the elements in the form.

2 lines of code and your entire form is bundled and ready to be uploaded.

Important

You must give each of your form input, select, and textarea elements a name attribute. Without the name attribute it will not get added to the FormData object.

You should always do validation on the data and not blindly upload it.

Now, you can also use the FormData object to bundle data that is not part of a form, or a mix of form data and other variables.

let fd = new FormData();
fd.append('name', 'value');
fd.append('email', document.getElementById('email').value);

This FormData object with the two name-value pairs can now be passed to the body property of a Request or options object for a fetch call.

URLSearchParams

The URLSearchParams object is the object that can be used to hold or build a QueryString. It can also be used as an alternative to FormData when bundling data to upload to the server. It can be used as part of the URL, in the headers, or in the body.

MDN reference for URLSearchParams (opens new window)

It is named as URLSearchParams because it represents the value held in the search property of the Location object, which is the top-level object that holds all the information about the webpage's url, hash, protocol, port, domain, path, etc.

MDN Location reference (opens new window)

It works much the same way as a Headers or FormData object do.

let search = new URLSearchParams();
search.set('key', 'value');
search.set('name', 'Karim');
search.has('key'); //true
search.get('name'); //Karim
search.sort(); //sorts the existing values by their key
search.forEach((val, key) => console.log(key, val));
search.delete('key');
search.toString(); //gives you the full string (without a `?`)

There is also an append method that works like set but will allow for duplicate entries with the same key.

It is an iterable object, which means it can be used in a for...of loop.

When you want to add a URLSearchParams string to the URL or to the body, use the toString method. Remember to add the ? in front of it if you are using it as part of the URL.

USVString

A USVString is a Unicode Scalar Value String. Basically it is a string designed for efficient text processing. It uses UTF-16 instead of UTF-8 for the holding of string values. This means that code-points that need 16-bit values to be represented can be saved as a single character. Chinese characters and Emojis both fall into this category. JavaScript will internally handle the conversion between UTF-8 strings and UTF-16 strings when you use the USVString.

When you see that a USVString is being used for an API, you can just think of it as a string that is encoded to work well in fetch calls and URLs.

Fetch All Together

Now that you know all the parts of an HTTP Request, you can build your own Request object and pass it to a fetch call.

The following example shows how to combine all the different parts into a Request object that gets passed to the fetch method. Only one value is being appended or added to the Headers, FormData, or URLSearchParms object for brevity's sake.

let head = new Headers(); //`append` what you need
head.append('x-custom-header', 'Memento');

let fd = new FormData(); //`append` what you need
fd.append('name', 'value');

let search = new URLSearchParams(); //`set` what you need
search.set('api-key', 'some-value');

let baseURL = `https://www.example.com`;
let relativePath = `./api/endpoint`;
let url = new URL(`${relativePath}?${search}`, baseURL);
// or new URL(path+querystring, baseurl).toString();
//call the toString method to create the DOMString to pass to the Request object

let req = new Request(url, {
  headers: head,
  method: 'POST',
  mode: 'cors',
  body: fd,
});

fetch(req)
  .then((response) => {
    //response to the fetch
  })
  .then()
  .catch();

CORS

Cross-Origin Resource Sharing (CORS) is the name of a set of rules that Browsers use in determining whether assets from other origins or domains are safe to downloaded or use.

There are Request headers and Response headers that are only allowed to be set by the browser or the server. There are others that we are allowed to dynamically create or edit with JavaScript.

Sometimes a file can be fetched but we are not allowed to use the contents.

There are special requests called preflight requests that are used to determine whether these rules are met.

Full explaination of CORS. A summary of what CORS is can be found in the video below.

Content-Security-Policies

The Content-Security-Policy is a security feature in the browser that lets you add to the headers or the <meta> tags in your document head, a list of the locations that the website is allowed to use to find all the different parts of the page. For example, if you had one place for fonts, two places for images, three allowed servers for scripts, and one allowed server for fetch calls, then all that can be defined in the Content-Security-Policy header.

CSP official site (opens new window)

Authentication and JWT

When you need to do things like pass a username and password along with a fetch request, there are a few things you can do.

The following videos will be very useful to you later on when you start using JWT in MAD9124 and need to accept tokens on the client side.

Tokens has become the most common way to authenticate with fetch and API calls. This first video explains how tokens can be passed through fetch.

This next video shows the full life cycle of JWT (JavaScript Web Tokens) on the client and server. The Server-side part of this will be discussed in detail in MAD9124.

What to do this week

TODO

Things to do before next week

• Read and Watch all the course content for 5.1, 5.2, and 6.1
• Start working on Hybrid Three