Data Structures

• If we have a good list implementation, do we need any other data structures?
• For computing: no
  • We can compute everything with just lists (actually even less). The underlying machine memory can be thought of as a list
• For thinking: yes
  • Lists are a very limited way of thinking about problems

List Limitations

• In a list, every element has direct relationships with only two others: the predecessor and the successor
• Access time: Θ(n)
• Goal: Θ(log n)

Complex Relationships: Phylogenetic Tree

Complex Relationships: Language Tree

List → Tree

• List: each element has relationships with up to 2 other elements:
• Binary Tree: each element has relationships with up to 3 other elements
  • A tree is a special case of a list

Tree Terms

• Root: a node with no parent; there can only be one root
• Leaf: a node with no children
• Siblings: two nodes with the same parent
• Height of a node: length of the longest path from that node to a leaf
  • Thus, all leaves have height of zero
  • Height of a tree: maximum depth of a node in that tree = height of the root
• Depth of a node: length of the path from the root to that node

More Tree Terms

• Path: sequence of nodes n₁, n₂, ..., n_k such that n_i is parent of n_i+1 for 1 ≤ i ≤ k
• Length: number of edges in the path
• Internal path length: sum of the depths of all the nodes

Other Examples of Trees

Files and folders on a computer Compilers: parse tree a = (b+c) * d; Genealogy trees These become complicated with some complex family relationships Lab 5: expression trees

First child/next sibling

class TreeNode { private: string element; TreeNode *firstChild; TreeNode *nextSibling; public: // ... }

Traversals of trees

• Pre-order: / * + 1 2 - 3 4 * 5 6
• In-order: (((1+2) * (3-4)) / (5*6))
  • Parentheses are added to handle operator precedence issues
• Post-order: 1 2 + 3 4 - * 5 6 * /

Pre-order Traversal

• Pre-order: node first, then children (this is pseudo-code):

  TreeNode::printTree() {
    this.print();
    for each child c of this:
        c.printTree();
  }
  

In-order Traversal

• In-order: left node first, then self, then right node (C++):

  void BST::print(BinaryNode *curNode) {
    if (curNode != NULL) {
        print(curNode->left);
        cout << curNode->element << endl;
        print(curNode->right);
    }
  }
  

Post-order Traversal

• Post-order: children first, then node (partly C++, partly pseudo-code):

  int TreeNode::numNodes(TreeNode *tnode) {
    if ( tnode == NULL )
        return 0;
    else {
        sum=0;
        for each child c of tnode
            sum += numNodes(c);
        return 1 + sum;
    }
  }
  

Traversals of trees (again)

• Pre-order: / * + 1 2 - 3 4 * 5 6
• In-order: (((1+2) * (3-4)) / (5*6))
  • Parentheses are added to handle operator precedence issues
• Post-order: 1 2 + 3 4 - * 5 6 * /

Binary Trees

All nodes have at most 2 children class BinaryNode { public: // ... private: int element; BinaryNode *left; BinaryNode *right; };

Binary Trees: diagram details

In reality, any child not shown is really a NULL pointer, as shown here; but these are generally omitted from the diagrams

Binary Search Trees (BST)

• Each node has a key value that can be compared
• Binary search tree property:
  • For a given node, which we will call the root...
  • Every node in left subtree has a key whose value is less than the root's key value, AND
  • Every node in right subtree has a key whose value is greater than the root's key value
• We assume that duplicate values are not allowed

BST: Example

BST: Counter-example

The difference

• Both binary trees and binary search trees have zero, one, or two children per node
• But a binary search tree is sorted
• However, most people, when they say "binary tree", really mean a "binary search tree"
• Note that we assume that we can NOT have duplicate elements in a BST

BST: find

• Basic idea:
  • Compare value to be found to key of the root of the tree
    • If they are equal, then done
    • If not equal, recurse depending on which half of tree the value should be in if it is in tree
    • If you hit a NULL pointer, then you have "run off" the bottom of the tree, and the value is not in the tree

BST: Find

• Trying to find 3 will go, from the root, left → left → right
• Trying to find 6 will go, from the root, right → left → left
  • At that point, we have "run off" the bottom of the tree (via 7's left-child pointer, which is NULL), and thus the value is not in the tree

BST: find

(no external source code)

BinaryNode * BST::find(int x, BinaryNode *curNode) {
    // handle case where a NULL pointer could be passed
    // curNode->right or curNode->left might be NULL
    if (curNode == NULL)  // we've "run" off the bottom
        return NULL;
    else if (x < curNode->element)
        return find(x, curNode->left);  // search left
    else if (x > curNode->element)
        return find(x, curNode->right); //search right
    else
        return curNode;    // matched
}

BST: insert

Do a find, and when we reach a NULL pointer, create a new node there

(no external source code)

void BST::insert(int x, BinaryNode * & curNode) {
    if (curNode==NULL)
        curNode = new BinaryNode(x,NULL,NULL);
    else if (x < curNode->element)
        insert(x, curNode->left);
    else if (x > curNode->element)
        insert(x, curNode->right);
    else
        ;    // duplicate... do nothing
}

BST: findMax(), findMin()

To find the maximum element in BST, traverse down the right subtree links Similarly down the left subttree links for findMin()

BST: remove

• Disrupts the tree structure
• Basic idea:
  • Find node to be removed
  • Three cases:
    • Node has no children
    • Node has one child
    • Node has two children

BST: remove: no children

• Just remove the node (reclaiming memory), adjusting the parent pointer to NULL
  • In this case, 9's left child link is changed to NULL

→

BST: remove: one child

• Adjust pointer of parent to point at child, and reclaim memory
  • In this case, 4's left pointer is changed to point to 3

→

BST: remove: two children

• Replace node with successor, then remove successor from tree
  • This requires running findMin() on the right sub-tree, and then removing that element
  • In this case, 5 is replaced by 7 (and the node that had 7 is removed)

→

BST Height

n-node BST: Worst case depth is n-1 This can easily happen if the data to be inserted is already sorted Claim: The maximum number nodes in a binary tree of height h is 2h+1-1

Proof by Induction on h

• Claim: the maximum number nodes in a binary tree of height h is 2^h+1-1
  • Or: n ≤ 2^h+1-1
• For h=0, tree has one node, which yields 2⁰⁺¹-1 = 1 nodes
• Assume the claim is true for any tree of height h
  • This would mean n ≤ 2^h+2-1 for a tree of height h+1
• Any tree of height h+1 has at most 2 subtrees of height h; each subtree has 2^h+1-1 nodes; add one more for the root
• Thus, our new tree of height h+1 has:
  • 2(2^h+1-1)+1 = 2^h+2-1 nodes
• If we put h+1 into our inductive hypothesis (instead of h), we get the same value; thus, it is proven

Relationship between h and n

• Given n nodes and height h, then by the claim (proven on the previous slide): n ≤ 2^h+1-1
• We can simplify:
  • n+1 ≤ 2^h+1
  • log₂(n+1) ≤ log₂(2^h+1)
  • log₂(n+1) ≤ h+1
• Thus h ≥ log₂(n+1)-1
• This means that the "shortest" tree we can achieve for n nodes is proportional to the base-2 log of the height

Perfect Binary Tree

• All leaves have the same depth
  • And all nodes have zero or two children, but not one
• Number of leaves: 2^h
• Number of nodes: 1 + 2 + 2² + 2³ + ... + 2^h = 2^h+1-1
• Problem: a perfect binary tree can only hold n values where n = 2^h+1-1
  • So you can't have, say, 5 values in a perfect binary tree!

Expression Trees

• A way to keep an internal representation of a mathematical equation
  • This allows a computer to compute the value of each node
• You will have to do this for lab 5
  • The lab expression tree only takes in integers as input; the more generalized version presented here includes variables as well
• See the Wikipedia article on Expression trees, specifically the section on construction of expression trees
  • That example is the same as the one here
  • Uses a stack!

Expression Tree Traversals

• Infix notation: (a + ((b+c) * d))
  • For our purposes, we will always place parentheses around all operations, even if they aren't needed
• Postfix notation: a b c + d * +
• Prefix notation: + a * + b c d

Building an expression tree

• Given an expression in postfix notation, the algorithm to bulid an expression tree is somewhat similar to that to evaluate a postfix expression (from lab 3)
• The algorithm is to read tokens from input:
  • If a number or a varaiable, push it on the stack
  • If an operator, pop off two values, attach them as children of the operator, and push that back onto the stack
• A proper postfix expression will have only one value in the stack upon completion

Example

Consider the postfix expression: a b + c d e + * *; the final expression tree is:

Example, step 1

• Expression: a  b + c d e + * *
• We read a, and push it onto the stack

Example, step 2

• Expression: a b + c d e + * *
• We read b, and push it onto the stack

Example, step 3

• Expression: a b + c d e + * *
• We read '+', pop off two values (a and b), put them as children of '+', and push the result back the stack

Example, step 4

• Expression: a b + c d e + * *
• We read c, and push it onto the stack

Example, step 5

• Expression: a b + c d e + * *
• We read d, and push it onto the stack

Example, step 6

• Expression: a b + c d e + * *
• We read e, and push it onto the stack

Example, step 7

• Expression: a b + c d e + * *
• We read '+', pop off two values (d and e), put them as children of '+', and push the result back the stack

Example, step 8

• Expression: a b + c d e + * *
• We read '*', pop off two values ('+' and c), put them as children of '*', and push the result back the stack

Example, step 9

• Expression: a b + c d e + * *
• We read '*', pop off two values ('*' and '+'), put them as children of '*', and push the result back the stack

The final result

Animation Tools

• A good AVL tree animation tool is here
  • A mirror that also contains the animation tool is here
• We'll be using this website throughout this slide set

AVL Trees

• Motivation: to guarantee Θ(log n) running time on find, insert, and remove
• Idea: Keep tree balanced after each operation
• Solution: AVL trees
  • Named after the inventors, Adelson-Velskii and Landis

AVL Tree Structure Property

For every node in the tree, the height of the left and right sub-trees differs at most by 1

AVL Tree

AVL balance factor

• Each node of a BST holds:
  • The data
  • Left and right child pointers
  • Possibly a parent node pointer
• An AVL tree also holds a balance factor
  • The height of the right subtree minus the height of the left subtree
    • We could have it be left minus right, but the convention in this class is to always have it be right minus left
  • Can be computed on the fly, as well, but that's VERY slow, and defeats the purpose of using AVL trees for speed

AVL tree balance

• "Balanced" trees
  • 0 means balanced
  • 1 means the right subtree is one longer than the left subtree
  • -1 means the left subtree is one longer than the right subtree
• "Unbalanced" trees
  • A balance factor of -2 or 2
  • We'll fix the tree
• Will we ever hit -3 or 3?

AVL Tree, with balance factors

By definition, a BST is a valid AVL tree if the balance factor for EVERY node is -1, 0, or 1

Not an AVL Tree

Not balanced: height difference greater than 1

AVL Trees: find, insert

• find: same as BST find
• insert: same as BST insert, except might need to "fix" the AVL tree after the insert (via rotations)
• Runtime analysis:
  • Θ(d), where d is the depth of the node being found/inserted
• What is the maximum height of an n-node AVL tree?

AVL tree operations

• Perform the operation (insert, delete)
• Move back up to the root, updating the balance factors
  • Why only those nodes?
  • Because those are the only ones who have had their subtrees altered
• Do tree rotations where the balance factors are 2 or -2

How many times to "fix" the tree?

• Any single insert will only modify the balance factor by one
  • So we fix the lowest off-balance nodes
  • Then everything above it is then balanced
• This means that we will have to only look at the bottom two unbalanced nodes

AVL insert

• Let x be the deepest node where imbalance occurs
• Four cases where the insert happened:
  1. In the left subtree of the left child of x
  2. In the right subtree of the left child of x
  3. In the left subtree of the right child of x
  4. In the right subtree of the right child of x
• Cases 1 & 4: perform a single rotation
• Cases 2 & 3: perform a double rotation

AVL single right rotation

→

• The node just inserted was node 1 (blue)
• The lowest node, immediately after the insert, with an imbalance is node 3 (red)
• Because node 1 is in the "left subtree of the left child" of node 3, this means we need to perform a single right rotation

AVL single left rotation

→

• The node just inserted was node 3 (red)
• The lowest node, immediately after the insert, with an imbalance is node 1 (blue)
• Because node 3 is in the "right subtree of the right child" of node 1, this means we need to perform a single left rotation

A side-effect of tree rotations

→

• This is the single right rotation
• Note that at least one node moves "up" (depth decreases)
  • In this case, nodes 1 and 2 both move up
• And at least one node moves "down" (depth increases)
  • In this case, node 3 moves down
• Similarly for a left rotation

AVL single right rotation: before & after

→

• Node 1 (red) is what is being inserted
• The lowest node with an imbalance is node 5 (balance: -2)
• Because the insert was in 5's "left subtree of the left child", we perform a single right rotation on 5 (and its left child, 3)

AVL single right rotation: before & after

→

• From the previous slide, we know we perform a single right rotation on 5 (and its left child, 3)
• Thus, the two blue nodes are the 'pivots' of the rotation
• Note that node 4 changes parents (from 3's right to 5's left)

AVL single right rotation: general case

→

$X < b < Y < a < Z$

The insert is into sub-tree X, increasing its height to h+1

Notice how sub-tree Y changes parent

Right and left rotations

Note that the trees shown are not necessarily AVL trees, but the rotations are correct

Cases 2 & 3: attempt a single rotation

→

$X < b < Y < a < Z$

The insert is into sub-tree Y, increasing its height to h+1

Failure! b's left subtree has height h+1; right is h+3

Double rotation

Node 5 (red) was just inserted The lowest node with an imbalance is node 8 (balance factor: -2) When discussing these rotations, we will call this the "parent" node Because the insert happened in 8's "right subtree of the left child", we perform a double rotation This consists of a single left rotation on the "child" (node 4), followed by a single right rotation on the "parent" (node 8)
Note that the two rotations are in different directions!

Double rotation, step 1

→

This is the single left rotation on the "child". The red node is what was inserted; the blue nodes are the 'pivots' of this single left rotation.

Double rotation, step 2

→

This is the single right rotation on the "parent". The red node is what was inserted; the green nodes are the 'pivots' of this single right rotation.

AVL double rotation: before & after

→

The red node is what was inserted

AVL double rotation: general case

→   $W < b < X < c < Y < a < Z$   The insert happens into X Notice sub-trees X and Y change parents

Ack! Terminology

• Some people will state a 'double left rotation'
  • But is that a left-right? Or a right-left?
  • We'll call them 'double left-right' and 'double right-left', which specifies the order to perform the operation on the child and then the parent

AVL insert, again

• Let x be the deepest node where imbalance occurs
• Four cases where the insert happened:
  1. In the left subtree of the left child of x
  2. In the right subtree of the left child of x
  3. In the left subtree of the right child of x
  4. In the right subtree of the right child of x
• Cases 1 & 4: perform a single rotation
• Cases 2 & 3: perform a double rotation

Algorithmic determination of rotation

left-left case	right-right case	left-right case	right-left case

• Given the lowest unbalanced node, and the child in the direction of the insert, compare the balance factors
• -2/+1 means a double left-right, +2/+1 means a singe left, etc.

All the tree rotations

AVL Tree: Runtime Analysis

• Find: Θ(log n) time: height of tree is always Θ(log n)
• Insert: Θ(log n) time: find() takes Θ(log n), then may have to visit every node on the path back up to root to perform up to 2 single rotations
• Remove: Θ(log n): left as an exercise
• Print: Θ(n): no matter the data structure, it will still take n steps to print n elements

Red-black trees

• Each node has a color attribute, which is either (wait for it...) red or black
• Animation site is here (mirror)

Red-black tree properties

All of these properties must hold for a red-black tree

1. A node is either red or black
2. The root is black
3. All leaves are black
   • The leaves may be the NULL children
4. Both children of every red node are black
   • Therefore, a black node is the only possible parent for a red node
5. Every simple path from a node to any descendant leaf contains the same number of black nodes
   • Counting or not counting the NULL black nodes; it doesn't make a difference as long as you are consistent

Insert

• Insert the node as for a normal BST
  • And color it red
• 5 possible cases:
  1. The new node is the root node
  2. The new node's parent is black
  3. Both the parent and uncle (aunt?) are red
  4. Parent is red, uncle is black, new node is the right child of parent
  5. Parent is red, uncle is black, new node is the left child of parent

Case 1: new node is root

• Case 1 is that the new node is the root node
  • Thus, it must be painted black, as per rule 2
  • This adds a black node to every path in the tree, so property 5 still holds

Case 2: new node's parent is black

• Case 2 is when the new node's parent is a black node
  • Property 4 (children of a red node are black) is not invalidated
  • Property 5 (all paths from a given node to its leaf nodes have the same number of black nodes) is not threatened
    • The new node's children are black, but that's the same number of black nodes as the node it replaced

Case 3: parent and uncle are red

• We change grandparent to red
  • And parent and uncle to black
• Property 5 still holds
• However, the grandparent may threaten property 2 or 4
  • So recursively do this on the grandparent
  • This occurs prior to any rotations

Case 4: P is red, U black, N is right child of P

• Perform a left rotation on P and N
  • Property 4 is still violated
• Now treat it as case 5 (where P is a left-child of N)

Case 5: P is red, U black, N is left child of P

• Perform a right rotation on P and G
• This will satisfy properties 4 and 5

Removal

• Do a normal BST remove
  • Find next highest/lowest value, put its value in the node to be deleted, remove that highest/lowest node
    • Note that that node won't have 2 children!
  • We replace the node to be deleted with its left child
    • This child is N, its sibling is S, its parent is P

Removal cases

• A total of 6 cases!
  1. N is the new root
  2. S is red
  3. P, S, and S's children are black
  4. S and S's children are black, but P is red
  5. S is black, S's left child is red, S's right child is black, and N is the left child of its parent
  6. S is black, S's right child is red, and N is the left child of parent P
• We won't see them in detail, though, but you can find details here

Why a red-black versus AVL?

• AVL trees are more rigidly balanced than red-black trees
  • Thus, more rotations are required during the operations in the worst case
• Time-critical applications will see a performance boost
• Functional programming languages used red-black trees for associative arrays (hashes)
  • The tree can be a persistent data structure
    • A data structure that retains a "memory" of its mutations

Splay Trees

• A self-balancing tree that keeps "recently" used nodes close to the top
  • This improves performance in some cases
  • Great for caches
  • Not good for uniform access
• Easier to implement than red-black or AVL trees

Main operation: splaying

• Anytime you find / insert / delete a node, you splay the tree around that node
  • Perform tree rotations to make that node the new root node
• Animation site is here (mirror)

Time complexity

• How long does a splay take?
  • Θ(h), where h is the height of the tree
• A splay tree can look like a linked list
  • Which means that the running time can be Θ(n) for a given operation!
  • That's not good...
• A splay tree is Θ(log n) amortized time

Amortized analysis

• An AVL tree will take Θ(log n) time for each and every operation in a series of m operations
  • Guaranteed
• A splay tree will take an average of Θ(log n) time for the same set of m operations
  • But some individual ones will take more time
  • So the total cumulative time to insert a set of elements will be the same for both types of trees

Splay tree conclusions

• Pros
  • Easy to implement
  • Θ(log n) amortized time
  • Good for caches where locality matters
• Cons
  • Not guaranteed Θ(log n) time for a single operation
  • Bad access time for uniform accesses

More amortized analysis

• Consider a vector insert
  • Worst case is that you have to double the size of the array (from n to 2n), and then copy the elements over, which takes n steps
• But this doesn't happen all that often!

Vector amortized running time

• Consider a vector of initial capacity of 100 elements, in which we insert about 1,600 elements
  • So n = 1600
• First "doubling" is from 100 → 200, costing 100 copies
  • Second is 200 → 400, costing 200 copies
  • Third is 400 → 800, costing 400 copies
  • Fourth is 800 → 1600, costing 800 copies
  • Assume we have not (yet) doubled from 1600 → 3200
• For 1,600 elements, we have 100+200+400+800 = 1500 copies

Vector amortized running time

• If we insert 3200 elements, we need 3100 copies
  • 6400 requires 6300 copies
  • n elements requires about n-100 copies
• Which means that the number of copies is proportional to n
• So a vector's insert is:
  • Θ(n) for the worst-case
  • Θ(1) amortized when executing many operations
• Although an individual operation may take n steps, a series of c operations will take a c*n steps
  • Where c is a constant
  • And when you divide by n, then each operation takes an average of c steps

When are trees not good to use?

• Trees are fast -- so when would we not want to use them?
  • When the items do not have a sorted order
    • A list of todo tasks
  • When we want less complexity
    • A stack or a queue
  • When we want an Θ(1) operation on retrieves
    • Vector get()
  • When we want an Θ(1) time for all operations
    • Hash tables can (almost) achieve that

Applications of Trees: Programs

• Any program can be represented as a tree; consider the following program (no external source code):

int z;

int foo (int x) {
    for ( int y = 0; y < x; y++ )
        cout << y << endl;
}

int main() {
    int z = 5;
    cout << "enter x" << endl;
    cin >> z;
    foo(z);
}

• Note that there are two int z declarations; this will be relevant shortly

The program tree

Notes on the program tree

• Called a "abstract syntax tree" or a "parse tree"
• Each node can be a different type
  • Having different properties and different number of children
    • A for loop node has four children (for init, for expression, for update, body)
    • A function node has at least three children (parameters, variables, body)
      • (we are ignoring other possible children of a function node here)
    • A body node has a variable number children
  • Done through inheritance!
    • All the specialty nodes inherit from a "node" class

Notes on the program tree

• Called an "abstract syntax tree" or a "parse tree"
• A compiler will build such a tree in memory
  • And trasverse it many times
  • For example, to figure out which 'z' is used in the main() function
  • Or to do code generation
    • Each node has an overridden method to generate the code for that node
  • Or to do type checking
  • Or to do code optimization

Comparing two programs

• What if we read in two programs...
  • ... and build parse trees for each
  • ... and compare their structure?
• We would be able to compare two programs while ignoring such things as:
  • Function/method order
  • Variable renaming
  • Different comments

Measure of Structural Similarity

• "A System for Detecting Software Plagiarism" (website)
  • The paper the site is based on can be found here
• It will load up all the programs for a class
  • I've run it in CS 101 with over 500 students
• And do all n² comparisons
  • That's 500² = 250,000 for CS 1110
    • Takes under 1 minute
  • Only 250² = 62,500 for CS 2150 this semester
    • Takes 10 or so seconds
• And display the most similar programs
• As an example, we'll use CS 101 (071) HW 4

The algorithm to implement

The results!

Because these results involve (anonymized) student work, they are not included in this repository...