CSI 312 Lecture 7, Fall 2011

ECE 313 - Computer Organization Lecture 25 - Pipelined Processor Design 1 2011 Reading: 6.1-6.2 2 Roadmap for the term: major topics  Overview / Abstractions and Technology  Performance  Instruction sets  Logic & arithmetic  Processor Implementation  Single-cycle implemenatation  Multicycle implementation  Pipelined Implementation   Memory systems  Input/Output 3 Pipelining Outline  Introduction  Defining Pipelining   Pipelining Instructions  Hazards  Pipelined Processor Design  Datapath  Control  Advanced Pipelining  Superscalar  Dynamic Pipelining  Examples 4 What is Pipelining?  A way of speeding up execution of instructions  Key idea: overlap execution of multiple instructions  Analogy: doing you laundry 1. Run load through washer 2. Run load through dryer 3. Fold clothes 4. Put away clothes 5. Go to 1  Observation: we can start another load as soon as we finish step 1! 5 The Laundry Analogy  Alice, Adel, Baysam, Dan each have one load of clothes to wash, dry, and fold  Washer takes 30 minutes  Dryer takes 30 minutes  “Folder” takes 30 minutes  “Stasher” takes 30 minutes to put clothes into drawers A B C D 6 If we do laundry sequentially... 30 T a s k O r d e r Time A 30 30 30 30 B 30 30 30 C 30 30 30 30 D 30 30 30 30 6 PM 7 8 9 10 11 12 1 2 AM  Time Required: 8 hours for 4 loads 7 12 2 AM 6 PM 7 8 9 10 11 1 Time 30 A C D B 30 30 30 30 30 30 T a s k O r d e r To Pipeline, We Overlap Tasks  Time Required: 3.5 Hours for 4 Loads  Latency remains 2 hours  Throughput improves by factor of 2.3 (decreases for more loads) 8 Pipelining a Digital System  Key idea: break big computation up into pieces  Separate each piece with a pipeline register 1ns 200ps 200ps 200ps 200ps 200ps Pipeline Register 9 Pipelining a Digital System  Why do this? Because it's faster for repeated computations 1ns Non-pipelined: 1 operation finishes every 1ns 200ps 200ps 200ps 200ps 200ps Pipelined: 1 operation finishes every 200ps 10 Comments about pipelining  Pipelining increases throughput, but not latency  Answer available every 200ps, BUT  A single computation still takes 1ns  Limitations:  Computations must be divisible into stage size  Pipeline registers add overhead 11 Pipelining Outline  Introduction  Defining Pipelining  Pipelining Instructions   Hazards  Pipelined Processor Design  Datapath  Control  Advanced Pipelining  Superscalar  Dynamic Pipelining  Examples 12 Pipelining a Processor  Recall the 5 steps in instruction execution: 1. Instruction Fetch 2. Instruction Decode and Register Read 3. Execution operation or calculate address 4. Memory access 5. Write result into register  Review: Single-Cycle Processor  All 5 steps done in a single clock cycle  Dedicated hardware required for each step  What happens if we break execution into multiple cycles, but keep the extra hardware? 13 Review - Single-Cycle Processor 5 5 16 RD1 RD2 RN1 RN2 WN WD Register File ALU E X T N D 16 32 RD WD Data Memory ADDR 5 Instruction I 32 M U X <<2 RD Instruction Memory ADDR PC 4 ADD ADD M U X 32 IF Instruction Fetch ID Instruction Decode EX Execute/ Address Calc. MEM Memory Access WB Write Back 14 Pipelining - Key Idea  Question: What happens if we break execution into multiple cycles, but keep the extra hardware?  Answer: in the best case, we can start executing a new instruction on each clock cycle - this is pipelining  Pipelining stages:  IF - Instruction Fetch  ID - Instruction Decode  EX - Execute / Address Calculation  MEM - Memory Access (read / write)  WB - Write Back (results into register file) 15 Basic Pipelined Processor IF/ID Pipeline Registers 5 5 16 RD1 RD2 RN1 RN2 WN WD Register File ALU E X T N D 16 32 RD WD Data Memory ADDR 5 Instruction I 32 M U X <<2 RD Instruction Memory ADDR PC 4 ADD ADD M U X 32 ID/EX EX/MEM MEM/WB 16 Single-Cycle vs. Pipelined Execution Non-Pipelined 0 200 400 600 800 1000 1200 1400 1600 1800 lw $1, 100($0) Instruction Fetch REG RD ALU REG WR MEM lw $2, 200($0) Instruction Fetch REG RD ALU REG WR MEM lw $3, 300($0) Instruction Fetch Time Instruction Order 800ps 800ps 800ps Pipelined 0 200 400 600 800 1000 1200 1400 1600 lw $1, 100($0) Instruction Fetch REG RD ALU REG WR MEM lw $2, 200($0) lw $3, 300($0) Time Instruction Order 200ps Instruction Fetch REG RD ALU REG WR MEM Instruction Fetch REG RD ALU REG WR MEM 200ps 200ps 200ps 200ps 200ps 200ps 17 Comments about Pipelining  The good news  Multiple instructions are being processed at same time  This works because stages are isolated by registers  Best case speedup of N  The bad news  Instructions interfere with each other - hazards • Example: different instructions may need the same piece of hardware (e.g., memory) in same clock cycle • Example: instruction may require a result produced by an earlier instruction that is not yet complete  Worst case: must suspend execution - stall 18 Pipelined Example - Executing Multiple Instructions  Consider the following instruction sequence: lw $r0, 10($r1) sw $r3, 20($r4) add $r5, $r6, $r7 sub $r8, $r9, $r10 19 Executing Multiple Instructions Clock Cycle 1 LW 5 RD1 RD2 RN1 RN2 WN WD Register File ALU E X T N D 16 32 RD WD Data Memory ADDR 32 M U X <<2 RD Instruction Memory ADDR PC 4 ADD ADD M U X 5 5 5 IF/ID ID/EX EX/MEM MEM/WB Zero 20 5 RD1 RD2 RN1 RN2 WN WD Register File ALU E X T N D 16 32 RD WD Data Memory ADDR 32 M U X <<2 RD Instruction Memory ADDR PC 4 ADD ADD M U X 5 5 5 IF/ID ID/EX EX/MEM MEM/WB Zero Executing Multiple Instructions Clock Cycle 2 LW SW 21 5 RD1 RD2 RN1 RN2 WN WD Register File ALU E X T N D 16 32 RD WD Data Memory ADDR 32 M U X <<2 RD Instruction Memory ADDR PC 4 ADD ADD M U X 5 5 5 IF/ID ID/EX EX/MEM MEM/WB Zero Executing Multiple Instructions Clock Cycle 3 LW SW ADD 22 5 RD1 RD2 RN1 RN2 WN WD Register File ALU E X T N D 16 32 RD WD Data Memory ADDR 32 M U X <<2 RD Instruction Memory ADDR PC 4 ADD ADD M U X 5 5 5 IF/ID ID/EX EX/MEM MEM/WB Zero Executing Multiple Instructions Clock Cycle 4 LW SW ADD SUB 23 Executing Multiple Instructions Clock Cycle 5 5 RD1 RD2 RN1 RN2 WN WD Register File ALU E X T N D 16 32 RD WD Data Memory ADDR 32 M U X <<2 RD Instruction Memory ADDR PC 4 ADD ADD M U X 5 5 5 IF/ID ID/EX EX/MEM MEM/WB Zero LW SW ADD SUB Lecture 17 - Pipelining 1 24 Executing Multiple Instructions Clock Cycle 6 5 RD1 RD2 RN1 RN2 WN WD Register File ALU E X T N D 16 32 RD WD Data Memory ADDR 32 M U X <<2 RD Instruction Memory ADDR PC 4 ADD ADD M U X 5 5 5 IF/ID ID/EX EX/MEM MEM/WB Zero SW ADD SUB Lecture 17 - Pipelining 1 25 Executing Multiple Instructions Clock Cycle 7 5 RD1 RD2 RN1 RN2 WN WD Register File ALU E X T N D 16 32 RD WD Data Memory ADDR 32 M U X <<2 RD Instruction Memory ADDR PC 4 ADD ADD M U X 5 5 5 IF/ID ID/EX EX/MEM MEM/WB Zero ADD SUB Lecture 17 - Pipelining 1 26 Executing Multiple Instructions Clock Cycle 8 5 RD1 RD2 RN1 RN2 WN WD Register File ALU E X T N D 16 32 RD WD Data Memory ADDR 32 M U X <<2 RD Instruction Memory ADDR PC 4 ADD ADD M U X 5 5 5 IF/ID ID/EX EX/MEM MEM/WB Zero SUB Lecture 17 - Pipelining 1 27 Alternative View - Multicycle Diagram IM REG ALU DM REG lw $r0, 10($r1) sw $r3, 20($r4) add $r5, $r6, $r7 CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 IM REG ALU DM REG IM REG ALU DM REG sub $r8, $r9, $r10 IM REG ALU DM REG CC 8 Lecture 17 - Pipelining 1 28 Pipelining Outline  Introduction  Defining Pipelining  Pipelining Instructions  Hazards   Pipelined Processor Design  Datapath  Control  Advanced Pipelining  Superscalar  Dynamic Pipelining  Examples Lecture 17 - Pipelining 1 29 Pipeline Hazards  Where one instruction cannot immediately follow another  Types of hazards  Structural hazards - attempt to use same resource twice  Control hazards - attempt to make decision before condition is evaluated  Data hazards - attempt to use data before it is ready  Can always resolve hazards by waiting Lecture 17 - Pipelining 1 30 Structural Hazards  Attempt to use same resource twice at same time  Example: Single Memory for instructions and data  Accessed by IF stage  Accessed at same time by MEM stage  Solutions  Delay second access by one clock cycle, OR  Provide separate memories for instructions, data • This is what the book does • This is called a “Harvard Architecture” • Real pipelined processors have separate caches Lecture 17 - Pipelining 1 31 0 2 4 6 8 10 Time 12 IF ID EX MEM WB 14 IF ID EX MEM WB IF ID EX MEM WB IF ID EX MEM WB 14 Example Structural Hazard - Single Memory Memory Conflict Lecture 17 - Pipelining 1 32 Control Hazards  Attempt to make a decision before condition is evaluated  Example: beq $s0, $s1, offset  Assume we add hardware to second stage to:  Compare fetched registers for equality  Compute branch target  This allows branch to be taken at end of second clock cycle  But, this still means result is not ready when we want to load the next instruction! Lecture 17 - Pipelining 1 33 Control Hazard Solutions  Stall - stop loading instructions until result is available  Predict - assume an outcome and continue fetching (undo if prediction is wrong)  Delayed branch - specify in architecture that following instruction is always executed Lecture 17 - Pipelining 1 34 Control Hazard - Stall beq writes PC here new PC used here 0 2 4 6 8 10 12 IF ID EX MEM WB 16 add $r4,$r5,$r6 beq $r0,$r1,tgt IF ID EX MEM WB IF ID EX MEM WB sw $s4,200($t5) 18 BUBBLE BUBBLE BUBBLE BUBBLE BUBBLE STALL ECE 313 Fall 2006 Lecture 17 - Pipelining 1 35 Control Hazard - Correct Prediction Fetch assuming branch taken 0 2 4 6 8 10 12 IF ID EX MEM WB 16 add $r4,$r5,$r6 beq $r0,$r1,tgt IF ID EX MEM WB IF ID EX MEM WB tgt: sw $s4,200($t5) 18 ECE 313 Fall 2006 Lecture 17 - Pipelining 1 36 Control Hazard - Incorrect Prediction “Squashed” instruction 0 2 4 6 8 10 12 IF ID EX MEM WB 16 add $r4,$r5,$r6 beq $r0,$r1,tgt IF ID EX MEM WB IF ID EX MEM WB 18 BUBBLE BUBBLE BUBBLE BUBBLE tgt: sw $s4,200($t5) (incorrect - STALL) IF or $r8,$r8,$r9 ECE 313 Fall 2006 Lecture 17 - Pipelining 1 37 Control Hazard - Delayed Branch always executes correct PC avail. here 0 2 4 6 8 10 12 IF ID EX MEM WB 16 add $r4,$r5,$r6 beq $r0,$r1,tgt IF ID EX MEM WB IF ID EX MEM WB 18 Branch SLOT: and $r6,$r6,$r7 tgt: sw $s4,200($t5) IF ID EX MEM WB ECE 313 Fall 2006 Lecture 17 - Pipelining 1 38 Summary - Control Hazard Solutions  Stall - stop fetching instr. until result is available  Significant performance penalty  Hardware required to stall  Predict - assume an outcome and continue fetching (undo if prediction is wrong)  Performance penalty only when guess wrong  Hardware required to "squash" instructions  Delayed branch - specify in architecture that following instruction is always executed  Compiler re-orders instructions into delay slot  Insert "NOP" (no-op) operations when can't use (~50%)  This is how original MIPS worked ECE 313 Fall 2006 Lecture 17 - Pipelining 1 39 Data Hazards  Attempt to use data before it is ready  Solutions  Stalling - wait until result is available  Forwarding- make data available inside datapath  Reordering instructions - use compiler to avoid hazards  Examples: add $s0, $t0, $t1 ; $s0 = $t0+$t1 sub $t2, $s0, $t3 ; $t2 = $s0-$t3 lw $s0, 0($t0) ; $s0 = MEM[$t0] sub $t2, $s0, $t3 ; $t2 = $s0-$t2 ECE 313 Fall 2006 Lecture 17 - Pipelining 1 40 Data Hazard - Stalling 0 2 4 6 8 10 12 IF ID EX MEM 16 add $s0 ,$t0,$t1 STALL 18 sub $t2, $s0 ,$t3 IF EX MEM STALL BUBBLE BUBBLE BUBBLE BUBBLE BUBBLE BUBBLE BUBBLE BUBBLE BUBBLE $s0 written here W s0 WB $s0 read here R s0 BUBBLE ECE 313 Fall 2006 Lecture 17 - Pipelining 1 41 Data Hazards - Forwarding  Key idea: connect new value directly to next stage  Still read s0, but ignore in favor of new result  Problem: what about load instructions? ID 0 2 4 6 8 10 12 IF ID EX MEM 16 add $s0 ,$t0,$t1 18 sub $t2, $s0 ,$t3 IF EX MEM W s0 WB R s0 new value of s0 ECE 313 Fall 2006 Lecture 17 - Pipelining 1 42 Data Hazards - Forwarding  STALL still required for load - data avail. after MEM  MIPS architecture calls this delayed load, initial implementations required compiler to deal with this ID 0 2 4 6 8 10 12 IF ID EX MEM 16 lw $s0 ,20($t1) 18 sub $t2, $s0 ,$t3 IF EX MEM W s0 WB R s0 new value of s0 STALL BUBBLE BUBBLE BUBBLE BUBBLE BUBBLE ECE 313 Fall 2006 Lecture 17 - Pipelining 1 43 Data Hazards - Reordering Instructions  Assuming we have data forwarding, what are the hazards in this code? lw $t0, 0($t1) lw $t2, 4($t1) sw $t2, 0($t1) sw $t0, 4($t1)  Reorder instructions to remove hazard: lw $t0, 0($t1) lw $t2, 4($t1) sw $t0, 4($t1) sw $t2, 0($t1) Performance in Pipelining 45 Speed Up Equation for Pipelining pipelined d unpipeline Time Cycle CPI Pipeline stall CPI Cycle Time Ideal Ideal CPI Pipeline depth Speedup     pipelined d unpipeline Time Cycle Pipeline stall CPI Cycle Time 1 Pipeline depth Speedup    per Inst Stall cycles CPI pipelined Ideal CPI Average   For simple RISC pipeline, CPI = 1: 46 Performance  Speed Up  Pipeline Depth; if ideal CPI is 1, then: pipelined d unpipeline Time Cycle Pipeline stall CPI Cycle Time 1 Pipeline depth Speedup    CPU time= Seconds = Instructions x Cycles x Seconds Program Program Instruction Cycle CPU time= Seconds = Instructions x Cycles x Seconds Program Program Instruction Cycle ° Time is measure of performance: latency or throughput ° CPI Law: ECE 313 Fall 2006 Lecture 17 - Pipelining 1 47 Summary - Pipelining Overview  Pipelining increase throughput (but not latency)  Hazards limit performance  Structural hazards  Control hazards  Data hazards