Tag Maps Workshop, 01/2024, Alexander Dunkel

![IR](images/01_emojotagmap_zoo.png?01 "Map (Goal of Workshop)")

We'll create a Tag Map for **a self chosen area**, from spatially referenced Twitter, Instagram and Flickr posts (and emoji).

* These maps can be created for any scale and area, depending on data availability and a predefined granularity of information. 
* Some example maps on [maps.alexanderdunkel.com](http://maps.alexanderdunkel.com/). A paper ([Dunkel, 2015](http://dx.doi.org/10.1016/j.landurbplan.2015.02.022)) explains the conceptual and programmatic background. 
* check out some additional information provided on [my blog](http://blog.alexanderdunkel.com).
* also see the [tagmaps documentation](https://ad.vgiscience.org/tagmaps/docs/), which includes this tutorial

[![IR](images/01_emojotagmap_campus.png "Map (Goal of Workshop)")](https://www.flickr.com/photos/64974314@N08/28424941169/in/album-72157628868173205/)  
Emoji Tag Map TUD Campus

For Tag Maps, the general idea is that tags & emoji are  
* placed on the map according their (collective) area of use and 
* scaled based on the overall number of people using respective tags/emoji

**Tag Maps are scale dependent.**

In other words, data must be clustered again each time a new scale or area is explored.

## Clipping Data

<div style="width:80%;height:70%;margin:auto">
 <img class="plain" src="images/04_startClipGeo.gif"/>
</div>
To start open \01_ClipGeo\ClipGeo.exe and 
click on Load folder _once_.

<details><summary>Do not change the default path</summary>

<code>
...\01_Input\
</code>

</details>

See a Singleton-Error (e.g. The type initialized for ‘SingletonCreator’ threw an exception)? Or Tiles are not loading?
![IR](images/singleton.png)

ClipGeo requires [DotNetFramework](https://dotnet.microsoft.com/download/dotnet-framework).

Try the following workaround, before installing DotNetFramework:

* Close ClipGeo; Go to folder \01_ClipGeo\..

Check available .Net Versions with DotNetVersions/DotNetVersions.exe

.. and unzip ExtractForSingletonError_DOTNET40SQL.7z (e.g.) & Overwrite

<div style="width:100%;height:70%">
<img class="plain" src="images/04_export_02.gif"/>
</div>
Optionally provide an *output name* before exporting data.

![IR](images/04_review.gif) 
Output data is stored in /Output/01_Clipped_Data/

This data can be imported to ArcGIS (import XY-Data) or used as input to *\02_TagMapsPy* to create a Tag Map.

Tag Maps will be covered in [Part 2](#/14).

**To use your own clipped area**

First, export photo locations for a specific area 
and copy results 
* **from**: 
 `01_ClipGeo\02_Output\01_ClippedData\YourExport\*.csv` 
* **to**: 
 `02_TagMapsPy\01_Input\*.csv`

**To skip clipping of data**

Copy any Input data source (e.g.: `TR_Istanbul`):
* **from**: 
 `01_ClipGeo\01_Input\TR_Istanbul\*.csv` 
* **to**: 
 `02_TagMapsPy\01_Input\*.csv`

**To use the workshop sample area data for Großer Garten**

Select CSVs and copy:
* **from**: 
 `01_ClipGeo\02_Output\01_ClippedData\GrosserGarten_Sample_Clip\*.csv` 
* **to**: 
 `02_TagMapsPy\01_Input\*.csv`

## Prepare Tagmaps environment

Open conda prompt inside folder `\02_TagMapsPy `

---

Create a "tagmaps" environment and install packages:
```bash
conda config --set channel_priority strict
conda create -n tagmaps --file tagmaps.txt
conda activate tagmaps
```

<details><summary>Environment exists?</summary>
Use another name (<code>-n</code>), e.g. <code>tagmaps_mary</code>
</details>

Note that <code class="highlight">conda_here</code> shortcut is prepared for the pools at TUD

<details><summary style="cursor: pointer;">Start conda at home?</summary>
Three steps:
<ul>
<li>Install <a href="https://docs.conda.io/en/latest/miniconda.html">Miniconda (package manager) </a></li>
<li>Open conda prompt (e.g. Windows: Start Menu > "Conda")</li>
<li>Change folder to <code class="highlight">01_Tools/02_TagMapsPy</code> (e.g. <code>cd ..</code>)</li>
</ul>
</details>

Create a "tagmaps" environment and install the package directly:
```bash
conda create -n tagmaps # creates an empty environment
conda activate tagmaps # activates the environment
conda config --env --set channel_priority strict # sets env options
conda install tagmaps --channel conda-forge # installs tagmaps package from conda-forge
```

The recommended way to install tagmaps, see [the docs](https://ad.vgiscience.org/tagmaps/docs/quick-guide/).

## Part 3: Visualization in ArcGIS

---

Go to folder 03_MappingArcGIS/ and open the template file.

<details><summary>Using ArcMap 10.3 to 10.6?</summary>
The slides have been updated for ArcGIS Pro 2.5. You can
view instructions for old versions of ArcMap [here](https://ad.vgiscience.org/tagmapsworkshop_tud/#/22).
</details>

For mapping Emoji, make sure you have a suitable font installed prior to opening ArcGIS. The font *TwitterColorEmoji* is used in 
the mxd and is available [here](https://github.com/eosrei/twemoji-color-font/releases).

(this _may_ not be necessary for Windows 10, which has native Emoji support)

To install TwitterColorEmoji:

1. Download [TwitterColorEmoji-SVGinOT-Win-*.*.*.zip](https://github.com/eosrei/twemoji-color-font/releases)
2. Extract it somewhere
3. Open `cmd` with elevated permissions (Start > Type `cmd` > run in Administrator Mode)
4. Cd into the folder where you have extracted the zip, e.g.: 
   ```
   cd C:\temp\TwitterColorEmoji-SVGinOT-Win-13.0.1
   ```
5. run install.cmd by typing `install.cmd`
6. the tool will compile Segoe UI Symbol for your system, and offer you to install the two ttf files
7. Type `Y` and install the two fonts

This will make sure you have the proper font installed. Segoe UI Symbol should be available by default, 
but it `a)` doesn't include all emoji and `b)` there may be something wrong with the font on your system.

The following slide shows a video of the process.

---

Your data should be automatically linked from `02_TagMapsPy/02_Output`

Important: Do not use `Data View` in ArcMap, use the (pre-set) **Layout View** (see [why](#/35/3))

Zoom to area under investigation, based on the extent of our clustered shapefile.

Check data links. This red ! is there deliberately, do not fix: 
![IR](images/redx.png)

These red !'s are not ok (and shouldn't happen): 
![IR](images/redx_fix.png)

View instructions below to fix.

A red "!" means links are missing, fix first:
<div style="width:100%;height:70%">
 <img class="plain" src="images/09_fixlinks_02.gif"/>
</div>

Do not use `Data > Repair Data source`, since this will remove any visualization rules contained in data layers.

Fixing links by `Properties > Source > Set Data Source` will leave any special layer settings untouched and simply repair the reference to underlying shape data.

All 3 Layers (e.g. *Top10*, *Other*, *HImp*) link to the same shapefile (*allTagCluster.shp*). Each layer has different queries defined, so that different classes of data can be visualized slightly different, 
e.g. Top10Layer:

"Weights" >= 300 AND "HImpTag" = 1

![IR](images/01_emojotagmap_zoo.png?01)

To view emoji-clustering, follow additional steps below.

There's a bug in [Fiona/GDAL](https://github.com/Toblerity/Fiona/issues/549) that currently prevents ArcGIS displaying Emoji-Unicode that is directly written to Shapefiles from Python.

Until this bug is solved, you can import Emoji through CSV and Join to TagCluster Shapefile. Follow steps below..

For mapping Emoji, make sure you have a suitable font installed prior to opening ArcGIS. The font *TwitterColorEmoji* is used in 
the mxd and is available [here](https://github.com/eosrei/twemoji-color-font/releases).

In Windows 10, the default emoji font "Segoe UI Symbol" will otherwise be used (but: poor emoji coverage).

Depending on the emoji font used, your map will look slighlty different:  
![IR](images/emoji-comp.png)

![IR](images/font_replace_02.gif)  
Optional: To change the emoji font, go to Layer Properties > Labels > Select Class "Emoji" and edit the expression.

![IR](images/11_emojiAdd_02.gif) 
Add allTagCluster.shp to ArcGIS.

![IR](images/11_emojiTableAdd_02.gif) 
Add emojiTable.csv to ArcGIS.

We'll now join the `emojiTable.csv` to the Tag Cluster Shapefile (based on FID)

![IR](images/11_emojiTableJoin_02.gif)

We'll now select all Clusters that are Emojis (Emoji-Column = 1)
![IR](images/11_emojiTableSel_02.gif)

Open Table and Copy Emojis from Join to Column "ImpTag"
![IR](images/11_emojiTableCopy_02.gif)

Remove Join and Polygon Layer
![IR](images/11_emojiTableFinal_02.gif)

Enable Emoji Tag Map Layer
![IR](images/11_emojiTableFinal_Zoom_02.gif)

[![IR](images/01_emojotagmap_campus_Loc.png)](https://www.flickr.com/photos/64974314@N08/40797729021/in/dateposted-public/)

To add an additional layer showing overall location clustering, 
view steps below.

Visualizing post clusters (e.g. photo locations) in background helps assessing overall frequentation patterns.

The data for this layer is generated next to Tag Clusters and can be found in `/Output/allLocationClusters.shp`.

![IR](images/21_enable_photoLocations_02.gif)  
Enable Location Cluster Layer: shows base clustering.

We'll analyze the patterns of photo clustering using the `Getis-Ord Gi* Statistic` that is available in Spatial Analyst Extension.

The results are used to colorize clusters in hot and cold spots.

**Optional Step: ISA**

For the Getis-Ord Statistic, we can provide a value for `Distance Threshold`.

We can use the Incremental Spatial Autocorrelation (ISA) tool to find a suitable Distance Threshold for our data.

The ISA tool visualizes clustering of data at different scales. Its output curve can be used to find a distance, where the clustering is "most pronounced" for the given area and data.

![IR](images/isa_02.gif) 
Let's see: Find the tool under `Spatial Statistics Tools/Analyzing Patterns`.

* Input Features: Add `AllLocationCluster.shp`
* Input Field: `Weights`
* Select a location to store `Output Table` and `Output Report File`

Click Run.

Open the results from ISA stored as PDF:  
![IR](images/isa2_02.gif)

Identify the first peak of your curve (from left to right).

Sometimes, there is no peak, sometimes there is more than one peak. If one or several peaks are shown, a suitable Threshold Distance is usually at the first shown peak.

![IR](images/isa_results.png) 
In the example above, a suitable Threshold Distance would be at around 187 Meters. Remember this (your) value for the next step.

![IR](images/21_hot_spot_02.gif) 
Find the Hot Spot Analysis (`GI* Statistic`) tool under `Spatial Statistic Tools/Mapping Clusters`.

![IR](images/HS250.gif)  
Add Location Clusters, select *Weights* field, optionally add a Distance Threshold (from ISA), specify output location, and click **Run**

![IR](images/LoadSymbol_02.gif) 
Load Symbology (double-click layer, Symbology: import from `Layer HS250_Sorted`). Activate "Update Ranges".

[![IR](images/FinalMap_02.gif)](https://www.flickr.com/photos/64974314@N08/40797729021/in/dateposted-public/)  
Sort layers and enable Location & Tag Clusters to view the final result.

![IR](images/21_symbol_size_02.gif)  
Optionally, adjust the formula for min/max symbol size to reduce size of smallest and largest cluster dots.

---

Optionally add or update:  
- Title  
- Legend  
- North Arrow, Scale Bar etc.

[![IR](images/GrosserGarten_Tagmap.png "Map (Goal of Workshop)")](images/GrosserGarten_Tagmap.pdf)  
Our final map with Tag, Emoji & Photo Location Clustering. Yay!

## Submit results

Please zip the following contents:
* log.txt
* allTagCluster.shp (the shapefile, all 5 parts)
* allLocationCluster.shp (the shapefile, all 5 parts)
* Basemap_World_ArcProV2.aprx (the ArcGIS Map)
* optional: exported PDF or PNG of map 
... as **firstname_lastname.zip** ..

.. and upload to [Opal Tag Maps folder](https://bildungsportal.sachsen.de/opal/auth/RepositoryEntry/3920855040/CourseNode/1642131038683050011)  
(or upload to [TU Cloudstore](https://cloudstore.zih.tu-dresden.de/) and send link to [alexander.dunkel@tu-dresden.de](mailto:alexander.dunkel@tu-dresden.de))

## Some evaluation criteria

* selected region:
    - is the map showing a number of interesting locations (good!) or is there only one big cluster that dominates all others? 
    - (not so good! ➡ select smaller region or different area)

* are globally irrelevant tags excluded?
    - e.g.: "Germany" can be considered of little interest to Dresden
    - ➡ should be removed from cluster analysis

* are single locations (e.g. Instagram "places") with unsuitable granularity excluded?
    - e.g. location "Dresden" is not suited to evaluate clustering at the city scale 
    - ➡ remove these locations prior to clustering

* bonus: 
 * scale bar and legend updated
 * data frame projection matches projection of shapefile
* don't take this too seriously: experiment!

## FAQ

A collection of frequently asked Tag Maps questions

Issue: Tag Map looks noisy, there're too many tags places on the map.

- often caused by too large area selection because there's a limit to the number of labels that can be placed on the map  
- either choose less variety of tags (e.g. 100 instead of 1000) or select smaller area

Issue: Tag Map shows one main hot spot surrounded by empty areas.

- this is a limitation of this type of visualization  
- in areas, where a single hot spot dominates, the Tag Map will simply illustrate this unequal distribution  
- either zoom in to hotspot and recluster or choose area with more equal distribution of locations

Issue: Zooming in ArcGis does not provide higher information resolution.

- the final maps are static and must be reclustered for each scale 
- this means that it is not possible to zoom in (e.g. in ArcGIS)
- therefore:
 - work in layout mode with locked extent, do not use data mode in ArcGIS 
 - reselect data (ClipGeo) and recluster for zooming in