EP3414737A1 - Autonomic system for determining critical points during laparoscopic surgery - Google Patents

Autonomic system for determining critical points during laparoscopic surgery

Info

Publication number
EP3414737A1
EP3414737A1 EP16872548.9A EP16872548A EP3414737A1 EP 3414737 A1 EP3414737 A1 EP 3414737A1 EP 16872548 A EP16872548 A EP 16872548A EP 3414737 A1 EP3414737 A1 EP 3414737A1
Authority
EP
European Patent Office
Prior art keywords
item
image
parameter
time
combination
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP16872548.9A
Other languages
German (de)
French (fr)
Other versions
EP3414737A4 (en
Inventor
Motti FRIMER
Tal Nir
Gal ATAROT
Lior ALPERT
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Transenterix Europe Sarl
Original Assignee
MST Medical Surgery Technologies Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by MST Medical Surgery Technologies Ltd filed Critical MST Medical Surgery Technologies Ltd
Publication of EP3414737A1 publication Critical patent/EP3414737A1/en
Publication of EP3414737A4 publication Critical patent/EP3414737A4/en
Withdrawn legal-status Critical Current

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/06—Devices, other than using radiation, for detecting or locating foreign bodies ; Determining position of diagnostic devices within or on the body of the patient
    • A61B5/065—Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe
    • A61B5/066—Superposing sensor position on an image of the patient, e.g. obtained by ultrasound or x-ray imaging
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
    • G16H20/40—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
    • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2048—Tracking techniques using an accelerometer or inertia sensor
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2051—Electromagnetic tracking systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2059—Mechanical position encoders
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2065—Tracking using image or pattern recognition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/30—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
    • A61B2090/309—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure using white LEDs
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2505/00—Evaluating, monitoring or diagnosing in the context of a particular type of medical care
    • A61B2505/05—Surgical care
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/06—Devices, other than using radiation, for detecting or locating foreign bodies ; Determining position of diagnostic devices within or on the body of the patient
    • A61B5/065—Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe
    • A61B5/067—Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe using accelerometers or gyroscopes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30241—Trajectory

Definitions

  • the present invention generally pertains to a system and method for providing a means and method for determining critical points in a procedure.
  • automatic detection of critical points in a procedure is needed in order to determine whether, for example, a procedure is complete, or whether an deviation has occurred in a procedure.
  • Automatic detection of critical points is also needed in order to identify surgical tasks in a surgical procedure, for example, in removal of a tumor from an organ, typical surgical tasks can include incising tissue around the tumor, grasping and removing the tumor, suction to remove blood and keep the surgical environment clear, cauterizing tissue, and suturing the incision.
  • Off-line identification of critical points such as the beginnings and ends of the surgical tasks can vastly improve efficiency in creating databases of surgical tasks and procedures which can be used for, among other things, training and assessing of surgical personnel and improving the quality of the procedures that are taught and used.
  • At least one processor in communication with said at least one imaging device; said at least one processor is configured to (z) analyze said at least one first image; (z ' z) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (Hi) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t, (z ' v) identify from at least one second image said at least one item, and, (v) calculate from said least one second image, for said at least one item, said at least one first parameter at said time t + At; and, c.
  • At least one communicable database configured to (z) store said at least one first parameter at said time t; and (z ' z) store said at least one first parameter at said time t + At; wherein, from comparison between said at least one first parameter at said time t and said at least one first parameter at said time t + At, said at least one critical point is identifiable.
  • said sensor is selected from a group consisting of an electromagnetic sensor; an ultrasound sensor; an inertial sensor to sense the angular velocity and the acceleration of the tool or other item; an accelerometer, a motion sensor, an EVIU, a sensor wearable by an operator, a sensor attachable to an item, an RFID tag attachable to an item, an ultrasound sensor, an infrared sensor, gyro-meter, tachometer, shaft encoder, rotary encode
  • At least one processor in communication with said at least one imaging device; said at least one processor is configured to (z) analyze said at least one first image; (ii) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (z ' z ' z) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t, (iv) compare said at least one first parameter at said time t to at least one prestored parameter at said time t, and, c.
  • At least one communicable database configured to (z) store said at least one first parameter at said time t; and (ii) store said at least one prestored parameter for any given time t; wherein, from comparison between said at least one first parameter at said time t and said at least one prestored parameter at said time t, said at least one critical point is identifiable.
  • At least one processor in communication with said at least one imaging device; said at least one processor is configured to (i) analyze said at least one first image, (ii) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (Hi) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t; (iv) identify from at least one second image said at least one item, and (v) calculate from said least one second image, for said at least one item, said at least one first parameter at said time t + At; and, iii.
  • At least one communicable database configured to (i) store said at least one first parameter at said time t; and; and (ii) store said at least one first parameter at said time t + At; b. providing, via said at least one imaging device, said at least one first image at said time t and said at least one second image at said time t + At; c. analyzing said at least one first image and determining, from said at least one first image, said at least one item; d. calculating from said at least one first image, for said at least one item, said at least one first parameter at said time t; e. storing said at least one first parameter at said time t; f.
  • At least one processor in communication with said i at least one imaging device; said at least one processor is configured to (i) analyze said at least one first image; (ii) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (Hi) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t, (iv) compare said at least one first parameter at said time t to at least one prestored parameter at said time t, and, iii.
  • At least one communicable database configured to (i) store said at least one first parameter at said time t; and (ii) store said at least one prestored parameter for any given time t; b. providing, via said at least one imaging device, said at least one image of a field of view at said time t; c. analyzing said at least one image and determining, from said at least one image, said at least one item; d. calculating from said at least one image, for said at least one item, said at least one first parameter at said time t; e. storing said at least one first parameter at said time t; f. analyzing said at least one second image and determining, from said at least one second image, said at least one item; and g. comparing said at least one first parameter at said time t to said at least one prestored parameter at said time t, thereby, from said comparison, identifying said at least one critical point.
  • Fig. 1 schematically illustrates an image of a surgical environment
  • Fig. 2 schematically illustrates movement of a tip of a surgical tool
  • Fig. 3A schematically illustrates speed of the tool tip and Fig. 3B schematically illustrates acceleration of the tool tip during the procedure;
  • Fig. 4A schematically illustrates speed of the tool tip,
  • Fig. 4B schematically illustrates acceleration of the tool tip and
  • Fig. 4C schematically illustrates jerk of the tool tip during the first part of the procedure;
  • Fig. 5 schematically illustrates critical points for a path
  • Fig. 6 schematically illustrates starting and ending point for a path
  • Fig. 7 schematically illustrates an ending point for a procedure
  • Fig. 8 schematically illustrates display overlay indicating an end-of-procedure
  • FIGs. 9 and 10 schematically illustrate an embodiment of a method of automatically assessing or automatically training an operator.
  • Fig. 11A-B depicts a robot arm including IMUs.
  • the term "fixed point” hereinafter refers to a location in a surgical field which is fixed relative to a known location.
  • the known location can be, for non-limiting example, an insertion point, a known location in a patient, a known location in an environment around a patient (e.g., an attachment point of a robotic manipulator to an operating table, a hospital bed, a wall of a room), a known location in a manipulation system, a known location in a practice dummy, or a known location in or on an operator.
  • a fixed point can also be a point marked in an image in a field of view of a surgical environment.
  • the term "item” hereinafter refers to any identifiable thing within a field of view of an imaging device.
  • An item can be something belonging to a body or something introduced into the body. Items also comprise things such as, for non-limiting example, shrapnel or parasites and non-physical things such as fixed points.
  • object hereinafter refers to an item naturally found within a body cavity. Non- limiting examples of an object include a blood vessel, an organ, a nerve, and a ligament, as well as an abnormality such as a lesion and a tumor.
  • tool refers to an item mechanically introducible into a body cavity.
  • a tool include a laparoscope, a light, a suction device, a grasper, a suture material, a needle, and a swab.
  • surgical object refers to a surgical tool, a robotic manipulator or other maneuvering system configured to manipulate a surgical tool, at least a portion of a light source, and at least a portion of an ablator.
  • a principal operator such as, but not limited to, the surgeon carrying out the main parts of the procedure
  • an assistant such as, but not limited to, a nurse
  • an observer such as, but not limited to, a senior surgeon providing instruction to or assessing a principal operator.
  • An identifier for an operator can include, but is not limited to, a name, an ID number, a function and any combination thereof.
  • identifiable unit refers to an identifiable purposive activity during a surgical operation, typically a minimal identifiable activity. Examples include, but are not limited to, movement of a needle and forceps to the site where a suture is to be made, making a knot in suture thread, activating fluid flow, and making an incision.
  • surgical task hereinafter refers to a connected series of at least one identifiable unit which comprises an identifiable activity.
  • surgical tasks that comprise more than one identifiable unit include, but are not limited to, making one suture, removing incised tissue from a surgical field, and clearing debris from a surgical field.
  • a non-limiting example of a surgical task that comprises a single identifiable unit is making an incision.
  • complete procedure refers to a connected series of at least one surgical task which forms an independent unit. For non-limiting example, closing an incision with one or more sutures will be referred to as a complete procedure.
  • a procedure can comprise tying the knot in a suture, making a single suture, or closing an incision with a series of sutures.
  • automated procedure hereinafter refers to a procedure in which one surgical tool automatically responds to a movement or other action of a second surgical tool.
  • Non-limiting examples of an automatically-executed procedure includes tracking a surgical tool by an endoscope or changing a lighting level in response to an increase in a perceived amount of smoke.
  • autonomic procedure refers to a procedure which can be executed independently of actions of a surgeon or of other tools.
  • Non-limiting examples of autonomic procedures include executing a complete suture and executing a plurality of sutures to close an incision.
  • a principal operator such as, but not limited to, the surgeon carrying out the main parts of the procedure
  • an assistant such as, but not limited to, a nurse
  • an observer such as, but not limited to, a senior surgeon providing instruction to or assessing a principal operator.
  • An identifier for an operator can include, but is not limited to, a name, an ID number, a function and any combination thereof.
  • aspects refer to a part of the outcome of a procedure, where the outcome is the overall result of the procedure.
  • aspects of an outcome include: amount of bleeding, speed of healing, return of an abnormality, occurrence of adhesions and patient discomfort.
  • the system of the present invention can determine critical points in a surgical procedure, either when the procedure is autonomously executed (without input from an operator) or when it is automatically executed (as an adjunct to an operator-executed procedure, such as moving an endoscope to ensure a continuously good view of suturing).
  • the system preferably analyzes at least one image of at least a portion of a field of view of the surgical environment, and by means of said analysis, determines a critical point, a characteristic movement, a characteristic position, a characteristic orientation and any combination thereof.
  • a plurality of critical points can comprise, for non-limiting example, a path for a surgical object to follow, an indication that a procedure has started (start of procedure), an indication that a procedure is complete (end of procedure) and any combination thereof.
  • At least one of the following can be determined:
  • the path can mark, for non-limiting example, movements of at least one surgical object during at least one procedure.
  • paths can mark movement of a needle and movement of a forceps during creation of a suture.
  • the path can mark the beginning and end of a line of sutures.
  • Critical points can mark, for non-limiting example, the locations of the sutures.
  • the path can be along the line of an incision, and would have to take into account the contours of the organ being sutured, possible movement of the organ, and ensuring that the suturing tool or tools can bypass any obstacles in the path.
  • a critical point for start of a procedure can be determined by comparing the movement of at least one surgical object to stored movements, by determining that the location of at least one surgical object matches the location of at least one fixed point and any combination thereof. If the movements of at least one surgical object match movements characterizing a start of a procedure, the procedure is deemed to be starting.
  • a critical point for an intermediate point in a procedure can be determined by comparing the movement of at least one surgical object to stored movements, by determining that the location of at least one surgical object matches the location of at least one fixed point and any combination thereof. If the movements of at least one surgical object match movements characterizing an intermediate point in a procedure, then an intermediate critical point has occurred.
  • an intermediate critical point can occur where one end of a suture thread passes under the other end and the operator must release a needle from a forceps and grasp it again.
  • a critical point for completion of a procedure can be determined by comparing the movement of at least one surgical object to stored movements, by determining that the location of at least one surgical object matches the location of at least one fixed point and any combination thereof. If the movements of at least one surgical object match movements characterizing an end of a procedure, the procedure is deemed complete.
  • the system can execute at least one of the following: •
  • a stored history can be tagged with at least one identifier, to enhance and simplify searching at least one library comprising at least one stored procedure.
  • a procedure can be tagged with an identifier of at least one operator, with a type of at least one procedure carried out, and with at least one characteristic of a patient. For non- limiting example, this could be used to determine the quality of outcome for appendectomies performed by Dr. Jones.
  • the system can interface with at least one surgical object, such as, but not limited to, a robotic maneuvering system, a surgical tool, a suction device, lighting, an ablator, a fluid supply and any combination thereof.
  • a surgical object such as, but not limited to, a robotic maneuvering system, a surgical tool, a suction device, lighting, an ablator, a fluid supply and any combination thereof.
  • the system determines that there is effusion of blood from an incision, it could command that a suction device be brought into the region of blood effusion and that suction be applied to the blood.
  • At least one fixed point can be marked so that the system can later be directed to move a surgical object to a selected fixed point, to move a surgical object to a known orientation, to set the surgical object into a known state, and any combination thereof.
  • the state of the surgical object can include, but is not limited to, a lighting level, a fluid flow level, a state of activated/deactivated, a known zoom, and a known temperature.
  • a path can take into account movement such as, for non-limiting example, movement of an organ.
  • the path can take into account a desired distance between two tools at the end of a path, for non-limiting example, if the path is that of a retractor separating tissue after an incision.
  • an artificial intelligence (AI) system analyzes at least one scene in at least one field of view and, from the analysis (and possibly from other provided information) forms an understanding of what is occurring. From this understanding, the AI system can identify at least one critical point and, preferably, can respond appropriately.
  • AI artificial intelligence
  • the system can calculate at least one performance metric and at least one parameter, and can derive at least one of an identifiable unit and a critical point.
  • Image-based tracking can identify the properties of at least one surgical object, including a surgical tool attached to a robotic arm, a surgical tool controlled directly by an operator, a static surgical object in a surgical environment and a surgical object connectable to a surgical object in the surgical field, as described herein.
  • this information can be used to avoid an obstacle, to instruct an operator to avoid an obstacle, to focus (for example, an endoscope) on a point of interest, to instruct an operator to focus on a point of interest, and any combination thereof.
  • the system can evaluate how an operator interacts with different objects to ascertain the operator's intent or identify the task that is currently being performed.
  • knot-tying identifiable unit can involve parameters such as total time spent tying a knot, idle time during knot tying, search time, approach time taken to reach the site of the knot, speed of the tools during knot tying, motion smoothness, bimanual dexterity, the length of the path followed by each tool, and the distance efficiency, which is a comparison of the actual path lengths with an optimal path length.
  • Both a parameter and an identifiable unit can be a function of time; either or both can change during a procedure.
  • any parameter or identifiable unit which can be a function of time is stored as a function of time.
  • the movements of at least one surgical tool held by an operator or moved by the system and the forces exerted by (and on) this surgical tool can be determined, from analysis of at least one image of at least a portion of the surgical environment, by a tracking subsystem, and any combination thereof.
  • Image analysis can be used to determine the location of at least one patient feature such as, but not limited to, an organ, a blood vessel, a nerve, a lesion, a tumor, a tissue, a bone, a ligament and any combination thereof.
  • Surgical tool identification and tracking and image analysis can be used to determine the location of at least one non-moving surgical tools such as a swab and also to determine a location of non-surgical items in the surgical environment.
  • Non-limiting examples of a non-surgical item in a surgical environment include such things as a glass shard or a bomb fragment.
  • Fig. 1 illustrates an image of a portion of a surgical environment (100). Both tools (110) and organs (120) are identifiable in the image.
  • the totality of the data on location, orientation and movements of surgical objects provides spatiotemporal 3 -dimensional data which characterize the surgical environment and the at least one item within it.
  • the spatiotemporal 3 -dimensional data can be analyzed in real time, stored in a database for later analysis and any combination thereof.
  • Other measureable and storable data include: grasping force, torsion about the tool axis, Cartesian forces, and any combination of these and the positions, orientations and movements disclosed above.
  • the system also can store at least one image of at least a portion of at least one procedure; this can also be synchronized with the force and position data.
  • Yet more measureable and storable data include: a distance between two items, a distance between an item and a fixed point, a distance between two fixed points, an angle between two items, an angle between an item and a reference line, an angle between two reference lines, an angle between at least one fixed point and at least one item, an angle between a reference line and at least one item, a volume between items, a volume between reference lines, a volume between at least one item and a reference line, a volume between at least one item and a fixed point, an angle between two items, and any combination thereof.
  • Table I gives a non- limiting example of metrics which can be assessed. In a given embodiment, any combination of metrics can be used.
  • Motion smoothness Motion analysis parameter based on the third time- derivative of position, which represents a change in acceleration
  • Transit profile 2D transit trajectory projected onto a plane Deviation on horizontal Deviation from the ideal course in the horizontal and plane and vertical plane vertical directions
  • EOA Economy of area
  • Interquartile force range Selects the 50 % of the data closest to the median so
  • Integral of the force Provides a measure of high forces and the amount of
  • Integral of the grasping Provides a measure of high forces and the amount of force time that forces are high.
  • Integral of the Cartesian Provides a measure of high forces and the amount of force time that forces are high.
  • Fig. 2 shows, schematically, the 3D movements, over time, of the tip of a surgical tool during a procedure.
  • Fig. 3A shows the speed of the tool tip during the procedure
  • Fig. 3B shows the acceleration of the tool tip during the procedure.
  • the speed, acceleration and jerk for the first part of the procedure are shown in Figs. 4A, B and C, respectively. From these, the metrics of Table II can be calculated.
  • Table II shows exemplary means of calculating the metrics of Table I.
  • the system of the present invention can comprise an advanced artificial intelligence (AI) system running on at least one processor which is capable of analyzing a scene in a field of view (FOV), as captured in real time by an imaging device and, from the analysis (and, in some embodiments, additional information from other sensors) forming an understanding of what is occurring. From this understanding, the system can derive at least one parameter, such as one of the metrics disclosed herein, and, from one or more parameters, determine at least one critical point in the surgical environment.
  • AI advanced artificial intelligence
  • the image captured by the imaging device can be a 2D image, a 3D image, a panoramic image, a high resolution image, a 3D image reconstructed from at least one 2D image, a 3D stereo image and any combination thereof.
  • a non-limiting example of a complete procedure comprising a number of surgical tasks, with each surgical task comprising an identifiable unit is suturing an incision, where the suturing process comprises a plurality of sutures.
  • the process of suturing an incision comprises a procedure consisting of a number of surgical tasks. Movement of at least one tool to the site of a first suture could be treated as a first surgical task. Each suture could constitute a surgical task; each movement of a tool from a completed suture to a next suture could constitute a surgical task, and movement of a tool away from a last suture could constitute a final surgical task.
  • the first surgical task would be moving at least one tool to a first critical point
  • the second would move at least one tool from the first critical point through a set of suturing movements to a second critical point
  • the movement from the first suture to the second suture is movement from the second critical point to a third critical point, which is the first critical point of the second suture, and so on.
  • a surgical tasks can be broken down into further surgical tasks or into identifiable units, and critical points assigned to each of the further surgical tasks or identifiable units.
  • creating a single suture can comprise several identifiable units, including at least one of the following: (1) inserting a needle through the tissue, (2) pulling suturing thread through the tissue, (3) grasping one end of the suturing thread with a grasper, (4) cutting one end of the thread, (5) passing one end of the thread around the other to make a first tie, (6) pulling the first tie tight, (7), passing one end of the thread around the other to make a second tie, thus forming a knot, (8) pulling the knot tight, (7) and (8) clipping short the thread ends, and (9) removing the clipped ends.
  • the surgical task of creating a suture would comprise a minimum of 18 critical points, namely, a critical point at the beginning and end of each identifiable unit.
  • Critical points can also be established at a point where there is a change in movement. For non-limiting example, movement of a tool from the location of a completed suture to the location of a next suture often involves a smooth movement upward and laterally away from both the completed suture and the tissue, followed by a downward, lateral movement to the location of the next suture.
  • a critical point can be established at the beginning of the upward, lateral movement, at the end of the smooth movement, where the speed and direction of movement change, at the beginning of the downward, lateral movement and at the end of the downward, lateral movement, where the speed of movement again changes.
  • FIG. 5 An idealized example of movement of a tool is shown in Fig. 5.
  • the movement of the tool is shown by the solid line (550), and an idealized movement of the tool is shown by the dashed line (560).
  • the dashed line is displaced downward slightly for clarity.
  • Each critical point (580) in the actual movement (550) indicates where the slope of the curve (550) changes.
  • the beginning of a procedure can be determined if at least one calculated first parameter is substantially equal to at least one prestored parameter.
  • a beginning of a procedure can similarly be determined if at least one calculated first parameter at a time t + At is substantially different from the same at least one first parameter at time t.
  • the beginning of the procedure occurs when the position, speed and acceleration of the tool (570) are substantially the same as the prestored parameters of position, speed and acceleration (575).
  • two first parameters or a first parameter and a prestored parameter are deemed to be different if the difference between the two is greater than a predetermined amount.
  • the predetermined amount can be in the range of about 0.1% to about 15%. In a preferred embodiment, the predetermined amount is about 5%.
  • the two first parameters or the first parameter and the prestored parameter are deemed to be substantially the same if the difference between the first parameter and the prestored parameter is no greater than the predetermined amount.
  • the exemplary tool movement shown in Fig. 5 can comprise a surgical task with a single beginning (570), a single end (590) and a number of critical points, or it can comprise up to 9 identifiable units, each comprising the minimum number, 2, of critical points.
  • a critical point can be established based on any of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, heating level in an ablator, amount of defogging, amount of smoke removal
  • the system identifies surgical tools in the working area; in preferred embodiments, objects such as organs, lesions, bleeding and other items related to the patient are identified, smoke, flowing fluid, and the quality of the lighting (level, dark spots, obscured spots, etc.). If smoke, flowing fluid or bleeding is identified, the system can respond by e.g., virtual smoke or fogging removal (removal of smoke of fogging from an image via software), increasing a lighting level, providing light from an additional direction or angle, starting smoke or fog removal measures such as flowing fluid across a lens or through an area obscured by smoke or fog, starting suction, alerting an operator to the bleeding, clarifying an image either in software or my changing zoom or focus, applying adaptive optic correction, and any combination thereof.
  • virtual smoke or fogging removal removal of smoke of fogging from an image via software
  • starting smoke or fog removal measures such as flowing fluid across a lens or through an area obscured by smoke or fog, starting suction
  • alerting an operator to the bleeding clarifying an image either in software or my changing zoom or focus, applying adaptive optic
  • a basis of a determination of the parameters from which critical points can be identified can be a determination of at least one first parameter of at least one item in the field of view.
  • This first parameter can be compared to either the first parameter at a different time or to a prestored parameter which is a second version of the same parameter, from a similar procedure executed at a different time.
  • the prestored parameter preferably stored in a database, can be an optimum or substantially optimum version of the first parameter, it can be a procedure executed by the same surgeon at a different time, a procedure executed by a different surgeon, a procedure executed on a simulator, a simulated procedure, and any combination thereof.
  • the prestored parameter can also be an average. The average can be generated by averaging together a plurality of procedures and extracting a prestored parameter from the averaged procedure, or the average can be generated by extracting the parameter for each of the plurality of procedures and averaging the parameters.
  • First and prestored parameters can be functions of time and, if so, are preferably stored as functions of time.
  • the at least one first parameter can be a 2D position, for non-limiting example, in the plane of the field of view, of at least a portion of the at least one item; a 2D orientation, for non- limiting example, in the plane of the field of view, of at least a portion of the at least one item; a 3D position of at least a portion of the at least one item; a 3D orientation of at least a portion of the at least one item; a 2D projection of a 3D position of at least a portion of the at least one item, a velocity of at least a portion of the at least one item, an acceleration of at least a portion of the at least one item, an angle of at least a portion of the at least one item, altering the state of at least a portion of the at least one item, and any combination thereof.
  • the movement of the at least one item can be selected from a group consisting of a maneuver of the item carried out by a robotic manipulator connected to the item, a movement of part of the item, a change in state of the item, and any combination thereof.
  • Non-limiting examples of movement of an item include displacing it, rotating it, zooming it, or, for an item with at least one bendable section, changing its articulation.
  • Non-limiting examples of movements of part of the item are opening or closing a grasper or retractor, or operating a suturing mechanism.
  • Non-limiting examples of a change in state of an item include: altering a lighting level, altering an amount of suction, altering an amount of fluid flow, altering a heating level in an ablator, altering an amount of defogging, or altering an amount of smoke removal.
  • two first parameters at different times and a first parameter and a prestored parameter are different when the difference between them is greater than 5%; parameters, either two first parameters or a first and prestored parameter are substantially the same if the difference between them is no greater than 5%.
  • a critical point including a critical point defining a beginning of a procedure or an end of a procedure can be determined if the difference between a first parameter at any given time t and the first parameter at a different time t + At is, greater than a predetermined value.
  • a critical point including a critical point defining a beginning of a procedure or an end of a procedure can be determined if the difference between a first parameter (that of the current procedure) and a prestored parameter (that of the stored procedure) is less than a predetermined value.
  • the critical point is determinable when the first parameter matches a prestored parameter tagged as a critical point.
  • a critical point, including a critical point defining a beginning of a procedure or an end of a procedure can be determined if the difference between a first parameter (that of the current procedure) and a prestored parameter (that of the stored procedure) is greater than a predetermined value.
  • At least the database and preferably both the tracking subsystem and the database are in communication with at least one processor configured to analyze the spatiotemporal 3- dimensional surgical database.
  • the result of the analysis can be determining a path for movement of at least one tool, determining a path start point, determining a path end point, identifying a start of a procedure, identifying a start of a feature, identifying an end of a procedure, identifying an end of a feature, determining a critical point, and any combination thereof.
  • At least one of a path, a critical point, the start of a procedure, and the end of a procedure can be provided as an indication to an operator.
  • the indication to the operator can be a visible signal (e.g., on a display), an audible signal, a tactile signal and any combination thereof.
  • the visible signal can be: a list of the identified at least one surgical procedure, a label indicating the at least one surgical procedure in a list of surgical procedures, and any combination thereof.
  • the label can be: a color, a pattern, an arrow, a line, a constant- brightness light, a variable-brightness light, a constant-color light, a variable-color light, a patterned light and any combination thereof.
  • the audible signal can be: a word, a constant-loudness sound, a variable-loudness sound, a constant-pitch sound, a variable-pitch sound, a patterned sound and any combination thereof.
  • the tactile signal can be a vibration, a constant-pressure signal, a varying-pressure signal, a stationary signal, a moving signal and any combination thereof.
  • the tactile signal can be applied to the head, the neck, the torso, an arm, a wrist, a hand, a finger, a leg, an ankle, a toe and any combination thereof.
  • the state of a surgical tool includes general properties such as its position, orientation, speed, and acceleration. It can also include tool-specific properties, such as location or relative location of a movable part of a surgical tool, such as, for non-limiting example, the locations of the faces of a gripper relative to each other (e.g., whether a gripper is open or closed), a lighting level, an amount of suction, an amount of fluid flow, a heating level in an ablator, an amount of defogging, an amount of smoke removal.
  • tool-specific properties such as location or relative location of a movable part of a surgical tool, such as, for non-limiting example, the locations of the faces of a gripper relative to each other (e.g., whether a gripper is open or closed), a lighting level, an amount of suction, an amount of fluid flow, a heating level in an ablator, an amount of defogging, an amount of smoke removal.
  • the system can identify at least one surgical tool in a surgical field, either by means of image recognition or by means of at least one tag associated with at least one surgical tool.
  • a tag can comprise color-coding or another mechanical label, or electronic coding, such as, but not limited to, radiofrequency signals.
  • a radiofrequency signal can be the same for different surgical tools or it can differ for at least one surgical tool.
  • the system can recognize a labelled tool from its mechanical or radiofrequency coding, a tool can be identified by an operator, and any combination thereof.
  • surgical tools comprise neither a marker nor a sensor, although at least one sensor can be used as part of at least one robotic tool-handling system.
  • the system determines position and orientation of at least one item via analysis of at least one image, preferably an image provided by a laparoscopic camera, and, preferably, at least a portion of the image is displayed and is visible to an operator.
  • the analysis of the image for non-limiting example, can be based on the shape of an item.
  • Item position and orientation can be determined, for example, via geometrical equations using the analysis of the projection of an item in the image plane or via Bayesian classifiers for detecting an item in the endoscopic image; the search for a surgical object can be further restricted by means of computing the projection of the surgical object's insertion point into the abdominal cavity.
  • Determination of item position and orientation in an image -based system can be rendered more difficult by ambiguous image structures, occlusions cause by blockage of the line of sight (e.g., by other items), blood, organs, smoke caused by electro-dissection, and any combination thereof.
  • Particle filtering in the Hough space can be used to improve the tracking process in the presence of smoke, occlusions or motion blurring.
  • Path calculation is applicable to manual, automatic and autonomic procedures.
  • a path can be overlaid on at least one image of a surgical environment, guiding an operator as to a path to be followed.
  • path calculation can be advantageous in predicting at least one movement of an operator and thereby increasing efficiency in following the operator's movements.
  • knowledge of an expected path can enable the system to configure a robotic arm's movement so as to prevent the arm from later being forced into an awkward or highly non-optimal orientation.
  • knowledge of a path can enable the fluid source and the endoscope to be maneuvered so that the fluid source remains in a good position for clearing the field of view and does not become "trapped" by the endoscope as both move from one ablation region to another.
  • path calculation can be used to suggest a path to an operator, for non-limiting example, if an actual path appears to the system to be highly non-optimal.
  • Knowledge that a procedure has ended can enable at least one of the following: determination of at least one next procedure to be carried out; a display of at least one next procedure, enabling an operator to select the next procedure; and autonomic selection of a next procedure.
  • Fig. 6 shows an illustrative, non-limiting example of finding a path, for an incision in a liver (610).
  • the starting point (640) and ending point (650) for an incision (620) are shown; the incision will divide the right lobe of the liver (610), while avoiding the falciform ligament (630).
  • the calculated path for the incision (620) is shown by a dashed line, which approximately parallels the falciform ligament (630).
  • Fig. 7 shows an illustrative, non-limiting example of determining the end of a procedure from the movements of tools.
  • the procedure is suturing to close an incision.
  • the locations of the sutures are schematically illustrated by the arcs (730, 740, 750).
  • the movement of the tip of the grasper holding the needle (710) and the movement of the tip of a second grasper (720) are shown.
  • the tools move diagonally upward away from the suture location, before moving diagonally downward to the location of the next suture.
  • an end to the procedure is determinable; movement of the graspers is nearly horizontally (760), instead of a diagonal rise.
  • a best practice or optimal procedure can be recorded from a procedure carried out by an experienced surgeon; it can be a compilation of fragments of recorded procedures, as carried out by several surgeons, thus combining the best parts a plurality of procedures to generate one best-practice procedure; or it can be a computer-generated procedure.
  • Fig. 8 shows an illustrative, non-limiting example of a displayed laparoscopic image (800) with an overlay (810) indicating the end of a procedure, in this case, the end of a suturing procedure.
  • Figs. 9-10 show an exemplary embodiment of a method of finding at least one critical point in a surgical environment.
  • the first step is to acquire at least one image of a field of view of an imaging device (910).
  • the image is analyzed (920), as described herein, to identify, in 3D, the position, orientation and movement of at least one tool and preferably all of the tools in the field of view, and, in some embodiments, the relationship of the at least one tool to the surgical environment (i.e., the organs, blood vessels, nerves, etc. of the patient) and, in some embodiments, the relationship of at least one tool to at least one other tool in the surgical environment.
  • the force exerted by or on at least one tool is also acquired.
  • At least one parameter of the tool is calculated (930). At least one of the parameters is then compared with a stored version of the same parameter (940) which, in some embodiments, is an optimum version of the parameter. If (950) the parameters are substantially the same, then no critical point has been found, and the method continues (circle 1) by acquiring (610) another image. If (950) the parameter(s) are not substantially the same, then a critical point has been found (960) and the method continues (circle 2) with Fig. 10.
  • a path is being identified, then (1020) a check is made whether all critical points in the path have been defined. If all critical points have been defined (1040), a path is calculated between the points. In preferred embodiments, the calculated path will avoid obstacles. After calculating at least one path, the routine ends. If not all critical points have been defined, an additional critical point is identified (1050), as a current location of at least a part of at least one tool, preferably a tool previously identified (950) as having at least one parameter different from the stored value of that parameter and the method continues (circle 1) by acquiring (910) another image.
  • the system of the present invention functions in real time or near real time.
  • any function or operation disclosed herein can be carried out either real time or offline at a later time.
  • a fixed point or critical point can be tagged manually.
  • Manual tagging can be by an operator indicating, by word, by gesture, or by touching a touchscreen, that a given point, such as the current position of a tool, is to be tagged as a critical point or as a fixed point.
  • automatic tagging can occur when a system identifies a point as a critical point or a fixed point.
  • semi-automatic tagging the system automatically identifies a point as a possible critical point or fixed point and provides an indication of the possible existence of a critical point or fixed point. The operator then manually confirms (or denies) the existence of the critical point or fixed point.
  • Distance, area, angle and volume can be determined from members of a group consisting of fixed points, critical points and any combination thereof.
  • Two or more points can define at least one distance.
  • Distance can be displayed as an individual, pair- wise distance, as a total distance, and any combination thereof.
  • a marker can indicate a point, a mark indicating the extent of distance, and any combination thereof.
  • Three or more points can define at least one angle.
  • Angle size can be displayed as an individual, trio-wise angle, as a total angle, and any combination thereof.
  • a marker can indicate a point, an edge, a mark indicating the extent of an angle, and any combination thereof.
  • Three or more points can also define an area; multiple areas can be defined.
  • a marker can indicate a boundary for an area, the size of an area, a selected area, a cumulative size for selected areas, and any combination thereof.
  • a marker can indicate a volume, the size of a volume, a selected volume, the cumulative size of selected volumes, and any combination thereof. Any combination of distance, angle, area and volume can be implemented. Any means known in the art of measuring distance, angle, area, volume known in the art can be implemented. Non-limiting examples of such measurement means include the methods typically found in Computer Aided Design (CAD) systems.
  • CAD Computer Aided Design
  • the distance marker can give the distance between the end points as a triple of values, typically the three distances (x, y, z) of a Euclidean coordinate system.
  • Other typical coordinate systems include, but are not limited to, cylindrical coordinate systems (r, ⁇ , z) and spherical coordinate systems (r, ⁇ , ⁇ ).
  • At least two fixed points can indicate a preferred path. Two fixed points can mark the beginning and end of a path; additional fixed points can mark locations in the path.
  • a path need not be a straight line.
  • a path can bypass obstacles.
  • a path can take into account non-flatness of, for example, an organ, if the path is to follow the surface of an organ.
  • a path can take into account movement such as, for non-limiting example, movement of an organ.
  • a path can take into account a desired distance between two tools at a point in a path, a desired orientation of at least one tool at at least one point in a path, and any combination thereof.
  • Non-limiting examples include: if the path is that of a retractor, the end points of the path can define the distance between the blades of a retractor; a path can be defined so as to prevent collision between a tool and an organ; a path can be defined so as to prevent collision between two tools; a path can be defined so as to follow the contours of an organ; a path can be defined to follow contours of a lesion; and a path can be defined so as to avoid an item such as a nerve or blood vessel.
  • a path can be stored in a database, can be displayed and any combination thereof.
  • a path is constrained to go through fixed points marking the path; in some embodiments, a path can deviate from fixed points in order to avoid an obstacle.
  • At least one path is calculated in 2D.
  • at least one path can be calculated in true 3D, enabling optimization of at least one of: the path, speed of movement of a surgical object during a procedure, position and orientation of a surgical object during a procedure, and avoiding collision with an item during a procedure. Any of the above can also be modified in real time. Such optimization and such modification can enable a surgical object to, for non-limiting example, avoid obstacles, move more slowly near items such as organs, prevent collision between tools, and any combination thereof.
  • Path calculation can also include use of information on a stage reached in a procedure and/or use of stored information on preferences of the operator, based on stored information about the methods of a particular operator.
  • sufficient depth information is provided so that the position and orientation of at least one item in the field of view can be determined in true 3D, enabling the accurate determination of a distance between at least two items, an angle between at least two items, an area between at least two items, a volume between at least two items and any combination thereof.
  • the 3D position and orientation of an item can be determined from image analysis, by using data from multiple cameras, from position sensors attached to tools, from position sensors attached to tool manipulators, from “dead reckoning" of tool positions and orientations coming from position and orientation commands to tool manipulators, and any combination thereof.
  • an accurate determination can be made as to whether a tool's position, orientation, speed, acceleration, smoothness of motion and other parameters are correct. It is also possible to determine if a tool is accurately following a desired path, whether a collision can occur between two items, and whether the distance between two items is small enough that one or both can be activated.
  • An item that can be activated or deactivated based on distance information can include, but is not limited to, an ablator, a gripper, a fluid source, a light source, a pair of scissors, and any combination thereof.
  • activation of an ablator is best delayed until the ablator is close to the tissue to be ablated so that heating does not occur away from the tissue to be ablated, to minimize the possibility of damage to other tissue.
  • the ablator can be automatically activated when the distance between the ablator and the tissue to be ablated is less than a predetermined distance, so that there is no unnecessary heating of fluid or tissue away from the tissue to be ablated and so that ablation is carried out efficiently.
  • an ablator could be activated when the 2D distance was small, but the distance perpendicular to the 2D plane (upward) was still large. In this case, the operator could be unaware of this until it was observed that the ablator was heating fluid rather than ablating tissue. The operator would then have to move the ablator downward until ablation could occur, but would not have, nor could he be given, information on how far downward to move. At this point, either the ablator could be deactivated and moved until it contacted the tissue, or the ablator could be left activated until ablation began. In either case, unwanted damage to the tissue is likely.
  • At least one distance marker is provided so that the operator knows the scale of the grid.
  • an orientation indication can be provided, a horizon can be marked and any combination thereof.
  • the orientation indication can be based on at least one item in a FOV such as, but not limited to, an organ, can be based on "dead reckoning", can be a direction relative to a region of interest, can be based on a position in a tool maneuvering system, can be based on a tool position determinable from a sensor signal, and any combination thereof.
  • Orientation can be determinable by providing a known orientation at a start of a procedure, by entering an orientation at a start of a procedure, by recognition of an orientation marker attached to a patient or to an operating table, and any combination thereof.
  • An orientation indication can allow an operator to remain aware of the orientation of a display view relative to a region of interest in the body, whatever the relative orientations of the body and the display view.
  • an orientation marker remains within a fixed region in the display view.
  • a non-limiting example of an orientation marker is axes of a 3D coordinate system, with the axes labeled so that the identity of each axis is discernable at a glance.
  • the axes are in a corner of the display view and rotate as the orientation of the display view changes.
  • an orientation marker comprises an arrow with a fixed center, the direction of the arrow indicating a fixed (3D) direction in space.
  • the point of the arrow will rotate around the center as the display view changes, while the color or texture of the arrow indicates whether the fixed direction is above or below the plane of the display image and the length of the arrow indicates the angle between the fixed direction and the plane of the display view.
  • At least one point in an FOV can be marked.
  • a markable point can be a critical point, indicate an organ or tissue, be a location on an organ or tissue, be a location within the body not on an organ or tissue, indicate a tool or other surgical object (such as a swab) introduced by an operator, or be a location (such as a tool tip) on a tool or other surgical object.
  • Sets of points such as but not limited to a set of critical points or a set of points forming the outline of an item or the surface of an item can also be marked.
  • a non-limiting example of an outline would be a line indicating the approximate extent of a tumor.
  • Marking can be by means of identifying a point in a display, which can be a 2D display or a 3D display; identifying a symbol representing an item, directing an indicator to a location by means of gestures or predetermined sounds, any other means known in the art of specifying a desired point, and any combination thereof.
  • Identification can be by means of touching a point or item, touching an image of a point or item, pointing at a point or item, pointing at an image of a point or item and any combination thereof.
  • a point can be labeled; with the point indicated in an image by a virtual marker.
  • a virtual marker can comprise any means of labeling images known in the art. Non- limiting examples of virtual markers include a predetermined geometrical shape, a predetermined word, a line encircling the image of a selected item, highlighting of the selected item (placing a patch of predetermined color or predetermined texture), and any combination thereof. Color-coding, with different colors indicating different types of virtual marker, can be used, either alone or in combination with any of the virtual markers described above.
  • a virtual marker can indicate a selectable display view.
  • selection of a marker automatically alters the display view to the view specified by the marker.
  • selectable display view markers can comprise, for non- limiting example, an outline of the selectable view, a point at the center of the selectable view, a patch or different color or texture covering the selectable view, and any combination thereof.
  • portions of the image are enhanced, typically in order to be seen or identified more easily.
  • An object which can be enhanced can include, but is not limited to, a blood vessel, a nerve, an organ, a ligament, a bone, a muscle, a lesion, a suspect location, a surgical tool, and any combination thereof.
  • enhancement can include, but is not limited to, increasing brightness, altering color, applying at least one color patch, applying at least one texture patch, applying a label, recoloring, and any combination thereof.
  • a marker can comprise a distance measurement, an angle measurement, an area measurement, a volume measurement and any combination thereof. Two or more points can define a distance; multiple distances can be selected, both contiguous and non-contiguous.
  • a marker can indicate a point, a distance, a path between points, a straight-line distance and any combination thereof.
  • a numerical marker can indicate a total path length, a total straight-line distance, at least one pairwise distance, and any combination thereof.
  • a critical point can be used for monitoring such as, but not limited to, safety monitoring.
  • Change in at least one critical point can indicate a change in performance, either of equipment or of an operator.
  • a change in performance related to the equipment can be flagged up and a procedure stopped or changed, or at least one correction can be applied to at least one movement to maintain a procedure within limits of safety.
  • an operator's performance can be monitored and at least one warning can be flagged up if an operator's performance falls below a predetermined level of safety.
  • At least a portion of at least one procedure can be stored as a record; the record can comprise a complete procedure or can be an edited portion thereof, typically a surgical task or an identifiable unit.
  • a record can comprise an image, an identifier, a fixed point, a critical point, a path and any combination thereof.
  • a record is tagged to enhance and simplify searching libraries of stored procedures, where a tagged identifier, image and any combination thereof is searchable.
  • a stored procedure can be a manually-executed procedure, an automatically-executed procedure, an autonomically-executed procedure and any combination thereof.
  • a complete procedure can comprise suturing an incision with a plurality of sutures.
  • a suturing procedure typically comprises a plurality of repetitions of the following identifiable units: (1) moving a suturing needle (held in a first grasper) and a second grasper to a location for a suture, (2) passing the suturing needle through the tissue, (3) using both graspers, tying a knot, and (4) cutting the suturing thread.
  • One record could comprise the complete procedure, the complete set of repetitions of steps (1) to (4); one record could comprise a surgical task, a single repetition of steps (1) to (4); one record could comprise an identifiable unit, a single one of the steps (1) to (4) and any combination thereof.
  • a parameter can be a 2D position, for non-limiting example, in the plane of the field of view, of at least a portion of the at least one item; a 2D orientation, for non-limiting example, in the plane of the field of view, of at least a portion of the at least one item; a 3D position of at least a portion of the at least one item; a 3D orientation of at least a portion of the at least one item; a 2D projection of a 3D position of at least a portion of the at least one item, a velocity of at least a portion of the at least one item, an acceleration of at least a portion of the at least one item, an angle of at least a portion of the at least one item, a state of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response
  • An identifier can include, but is not limited to, an identifier of an operator, type of procedure, a previous procedure during a surgical operation, a parameter, an identifier for an operating room, a physical characteristic of an operating room (e.g., temperature, humidity, time and date of cleaning, cleaning procedure, cleaning materials, type of lighting), a date of the procedure, a time and day of the week of a procedure, a duration of a procedure, a time from start of a previous procedure until start of a procedure, a time from end of a procedure until start of a subsequent procedure, an identifier of a patient, a physical characteristic of a patient, an outcome of a procedure, a type of malfunction during a procedure, severity of malfunction during a procedure, start time of malfunction, end time of malfunction, length of hospital stay for a patient, a readmission for a patient, and any combination thereof.
  • a non-limiting physical characteristic of a patient can include: age, height, weight, body mass index, health status, medical status, and any combination thereof.
  • Tagging can be manual or automatic. For non-limiting example, typically, names of operators will be entered manually. In another non-limiting example, a critical point or a fixed point can be tagged manually or automatically.
  • manual tagging can be by an operator indicating, by word, by gesture, or by touching a touchscreen, that a given point, such as the current position of a tool, is to be tagged as a critical point or a fixed point.
  • automatic tagging can occur when a system identifies a point as a critical point or a fixed point.
  • An outcome of a procedure can have one or more aspects.
  • an aspect include: amount of bleeding after completion of a procedure, amount of bleeding during a procedure, return of an abnormality such as a tumor, speed of healing, adhesions and patient discomfort.
  • a successful aspect would constitute: minimal bleeding after completion of a procedure, minimal bleeding during a procedure, no return of the abnormality, rapid healing, no adhesions and minimal patient discomfort.
  • a partially successful aspect would constitute: some bleeding after completion of a procedure, some bleeding during a procedure, minimal return of the abnormality, moderately rapid healing, a few small adhesions and some patient discomfort.
  • a partial failure in the aspect would constitute: significant bleeding after completion of a procedure, significant bleeding during a procedure, return of a significant amount of the abnormality, slow healing, significant adhesions and significant patient discomfort.
  • complete failure in the aspect would constitute: serious or life-threatening bleeding after completion of a procedure, serious or life-threatening bleeding during a procedure, rapid return of the abnormality, very slow healing or failure to heal, serious adhesions and great patient discomfort. It is clear that an outcome can include any combination of aspects.
  • a procedure could have minimal bleeding, both during and after the procedure (successful) with a few adhesions (partial success), but significant patient discomfort (partial failure) and rapid return of the abnormality (complete failure).
  • At least one stored record of at least one procedure can become part of at least one "big data" analysis
  • a big data analysis can be, for non-limiting example, for a procedure, for an operator, for a hospital or medical center, for a surgical object, for a robotic maneuvering system and any combination thereof.
  • the system can perform at least one procedure automatically, without an operator's intervention.
  • a non-limiting example of automatic control of a tool is control of zooming so that an operator has an optimum view during suturing.
  • the laparoscope adjusts position and focus without operator intervention; the laparoscope zooms in during the tying process, zooms out after a suture has been completed to allow a better overview of the site, and follows the suturing tools as they are moved to the site of a next suture.
  • the system can perform at least one procedure autonomically, without an operator's intervention.
  • a non-limiting example of autonomic functioning is an extension of the above, where the system carries out a suturing process without operator intervention, including moving the suturing tools, tying the sutures and cutting the suture threads, and moving an imaging device so that the process is observable.
  • an override facility is provided, so that an operator can intervene manually.
  • Manual intervention via a manual override, can occur, for non-limiting example, if an event occurs that requires immediate action.
  • the system can have different operation modes, depending on the identified procedure or the viewed scene.
  • the system identifies surgical tools in the working area; in some embodiments, objects such as organs, lesions, bleeding and other items related to the patient are identified, smoke, flowing fluid, and the quality of the lighting (level, dark spots, obscured spots, etc.). If smoke, flowing fluid or bleeding is identified, the system can respond by e.g., virtual smoke or fogging removal (removal of smoke of fogging from an image via software), increasing a lighting level, providing light from an additional direction or angle, starting smoke or fog removal measures such as flowing fluid across a lens or through an area obscured by smoke or fog, starting suction, alerting an operator to the bleeding, clarifying an image either in software or my changing zoom or focus, applying adaptive optic correction, and any combination thereof.
  • virtual smoke or fogging removal removal of smoke of fogging from an image via software
  • starting smoke or fog removal measures such as flowing fluid across a lens or through an area obscured by smoke or fog, starting suction
  • alerting an operator to the bleeding clarifying an image either in software or my changing zoom or focus, applying adaptive optic
  • software -based super-resolution techniques can be used to sharpen at least one image without physically changing zoom or focus of an endoscope.
  • At least one panoramic view is providable, either from a wide-angle lens or from unification of a plurality of images.
  • software correction can be applied to provide a user with an undistorted image of a field of view, or an image with a desired degree or type of distortion.
  • a panoramic view can provide at least a 180° view and preferably it can provide a substantially 360° view.
  • at least one image can be stabilized via software, the stabilization removing shake or other unwanted movement, thereby increasing clarity.
  • not all of the field of view of the lens is displayed.
  • either a wide-angle lens or a series of images of different regions of the operating field provide the system with a view of substantially all of an operating field.
  • a tracked object can be, but is not limited to: a surgical tool, an item of interest such as, but not limited to, bleeding, heart rate via pulsation of a blood vessel, unexpected movement of tissue, and any combination thereof.
  • the system can perform zooming, can move the center of the field of view, or can display an alert or a popup to show or to follow a tracked object outside of the displayed image, or to show or to follow an item of interest.
  • control of movement of a surgical object can include a member of a group consisting of: changing arm movement and trajectory according to the FOV, changing velocity of movement according to the amount of zoom, closeness to an obstacle or stage in a procedure, and any combination thereof.
  • a rule-based approach will be used to determine movement or changes thereof.
  • Images from other modalities such as, but not limited to, MRI images, CT images, ultrasound images, X-ray images, fluoroscopic images, molecular images, scintigraphy, SPECT, positron emission tomography (PET), other types of tomography, elastography, tactile imaging, photoacoustic imaging, thermography, Functional near-infrared spectroscopy (FNIR) and any combination thereof. Images from other modalities can be stored or realtime.
  • At least one device controllable by the system is bed-mounted. In preferred embodiments, this reduces the footprint of the system over the patient.
  • the system comprises control of a laparoscope.
  • the laparoscope has a wide-angle lens, preferably a high-definition lens.
  • the laparoscope is an articulated laparoscope; the system can comprise both a wide angle-lens and an articulated laparoscope.
  • the displayed field of view can be controlled by movement of the laparoscope, by virtual FOV control (computer control of the FOV by altering the displayed portion of the image), and any combination thereof.
  • at least one tool can be automatically tracked by the system.
  • the robotic arms are snake-like robotic arms providing full control by visual servoing (adaptive control via aimage analytics). This enables closed-loop control of all DOF's and, therefore, closed loop control of locating the target. Closed loop control also enables optimization by building an adaptive kinematic model for control of the robotic arms.
  • At least a portion of at least one surgical object can be tracked using the kinematics of the robot.
  • a 3D pose of a surgical object can be calculated using known or measured joint angles of the robot, the parameters of the system and a keyhole-based (due to the trocar point of surgical insertion) forward kinematics algorithm.
  • the system can be in communication with at least one other device or system.
  • the control software can control at least one surgical object, can be in communication with at least one advanced imaging system, can function as part of an integrated operating room and any combination thereof.
  • control software can have full connectivity with a member of a group consisting of: digital documentation, PACS, navigation, other health ⁇ systems, and any combination thereof.
  • At least one sensor can be used to provide information on location, orientation, speed, acceleration, and force exerted by or on at least one surgical tool.
  • a sensor (such as, for non-limiting example, at least one motion sensor) can be in mechanical communication with at least one tool; it (such as, for non-limiting example, a RFID tag) can be in wired communication with the tool; it can be in wireless communication with the tool; at least one processor can determine movement of at least one tool by determining change in position of at least one robot arm; and any combination thereof.
  • a sensor can be an accelerometer, a motion sensor, an EVIU, a sensor wearable by an operator, a sensor attachable to an item, an RFID tag attachable to an item, an ultrasound sensor, an infrared sensor, a CT image, an MRI image, an X-ray image, and any combination thereof.
  • one IMU can be mounted internally, in conjunction with a surgical tool such as a laparoscope, with a second IMU mounted on a movement-control robot, which improves the robustness of a determination of the position, orientation and movement of the surgical tool.
  • An item in a field of view of at least one image can be tracked by identifying at least one inherent, distinguishing characteristic of the item. For example, this can include the shape, color, texture, and movement of an item.
  • an item can be modified to make it more recognizable in the camera image. For instance, a colored marker, a tracking pattern and any combination thereof can be affixed to at least one surgical object to aid in detection of the surgical object by a computer algorithm or to aid in providing an instruction to an operator.
  • tracking technologies include sensor-based technologies, including optical sensor technologies, electromagnetic sensor technologies and ultrasound sensor technologies.
  • Optical techniques such as an infrared tracking system, can locate an object that has at least one infrared marker attached to it. The object being tracked does not require any wires, but the line of sight from the tracking system to the tracked object must be kept clear.
  • the senor can be, for non-limiting example, an electromagnetic sensor; an ultrasound sensor; an inertial sensor to sense the angular velocity and the acceleration of the tool or other item; a gyroscope, an accelerometer, an IMU and any combination thereof.
  • An electromagnetic tracking system can also be used to locate a surgical object.
  • At least one magnetic sensor can be affixed to a surgical object, and a magnetic transmitter emits a field that a magnetic sensor can detect.
  • a plurality of sensors can be combined into a single unit, such as an Inertial Measurement Unit (IMU).
  • IMU Inertial Measurement Unit
  • an IMU incorporates sensors such as an accelerometer, a gyroscope, a magnetometer and any combination thereof to track at least one of the orientation, position, and velocity of a surgical object.
  • An IMU can measure and report specific force, angular rate, and, if a magnetometer is included, magnetic field.
  • An IMU can be designed to transmit data wirelessly.
  • An IMU can be used to provide dead-reckoning control of objects.
  • IMUs can experience increasing error over time (especially in position), and some of the sensors can be sensitive to interference from other devices in an operating room.
  • two IMUs are used for dead reckoning, one at the base, the proximal end, of a flexible robotic arm and one at or near the distal end of the flexible robotic arm.
  • the distal IMU can be mounted to a laparoscope or other surgical tool or it can be mounted on the distal end of the flexible robotic arm itself.
  • other IMUs are provided at intermediate positions along the flexible robotic arm.
  • Fig. 11A schematically indicates an embodiment of a flexible robotic arm (500) which includes proximal and distal IMUs (510).
  • Fig. 11B schematically indicates an embodiment of a flexible robotic arm (500) with intermediate IMUs (515) in addition to the proximal and distal IMUs (510).
  • An optimum (second) parameter is typically derived from a previous procedure.
  • the procedure can be: a manually-executed procedure, an automatically-executed procedure, an autonomically-executed procedure, a simulation of a procedure and any combination thereof.
  • a manually-executed procedure was preferably executed by an experienced surgeon.
  • An automatically-executed procedure is one in which one surgical tool automatically responds to a movement or other action of a second surgical tool.
  • Non-limiting examples of automatically-executed procedures include tracking of a surgical tool by an endoscope or changing a lighting level in response to an increase in a perceived amount of smoke.
  • An autonomically-executed procedure is one which can be executed independently of actions of a surgeon or of other tools.
  • Non-limiting examples of autonomic procedures include executing a complete suture and executing a plurality of sutures to close an incision.
  • a simulated procedure can be a procedure carried out by an operator on a simulator or a procedure automatically or autonomically executed on a simulator or using simulated or stored data.
  • a prestored parameter can also be generated by averaging, either by averaging a plurality of procedures and extracting a single parameter from the averaged procedure, or by generating a parameter for each procedure and then averaging the generated parameters.
  • Kinematic tracking can be used to determine at least one property of at least one surgical object for a surgical object under robotic control.
  • a typical robotic system includes one or more jointed arms that can manipulate at least one surgical object on behalf of an operator.
  • the robot arm can also include at least one sensor (such as, but not limited to, an encoder, a potentiometer, a motion sensor, and an accelerometer) that can accurately determine the state of each joint.
  • the fixed properties of the physical structure of the robot arm are known (lengths of links, twists, etc.), they can be combined with the dynamic joint values to form a mathematical model of the robot arm. Desired surgical object properties, such as the position and orientation of the end-effector, the surgical tool, can be computed from this model, for non-limiting example, by forward kinematics.
  • Inverse kinematics can be used to determine, from the position and orientation of a surgical tool, the positions and orientations of the joint parameters that provided that position of the surgical tool.
  • an item can be tracked by identifying at least one inherent, distinguishing characteristic of the item.
  • the distinguishing characteristic can be the shape, color, texture, movement and any combination thereof of an item.
  • an item can be modified to make it more recognizable in an image. For instance, a colored marker, a tracking pattern, an LED and any combination thereof can be affixed to at least one surgical object to aid in detection by a computer algorithm or to aid in providing an instruction to an operator.
  • a minimum of three non-collinear markers is necessary for determining six DOF, if the sole means of determining surgical tool location and orientation is the marker.
  • a tracking system using at least one marker can provide high accuracy and reliability, it depends on a clear line of sight between the tracked tool and the camera system.
  • At least one LED can also be used to measure distance between a surgical object and tissue, typically by reflecting from tissue light emitted by an LED attached to a surgical object.
  • a tool can have a marker attached near its handle (outside the body) and a colored patch near its tip (inside the body).
  • movement of the marker is tracked by a camera outside the body (outside-outside), while movement of the colored patch is tracked by a camera inside the body (inside-inside).
  • an EM emitter can be close to the tip of the tool, while the EM sensor is attached to the operating table (inside-outside).
  • An electromagnetic (EM) tracking system can also be used to locate at least one surgical object. By computing the position and orientation of at least one electromagnetic receiver on a surgical object as it moves through space, a dynamic, real-time measurement of the position and orientation of a surgical object can be found.
  • EM electromagnetic
  • the electromagnetic receivers are attached to an operator's hands, tracking the movements (changing position and orientation) of the tools by tracking the movements of the hands.
  • keeping the sensors in a stable position during the entire execution of a surgical procedure can be difficult.
  • the movements of the surgeon's hands need not be directly related to movements of a surgical tool.
  • Electromagnetic tracking systems do not need a clear line of sight, but are strongly affected by ferromagnetic objects, such as steel tools or electronic equipment in the clinical environment, which can seriously degrade tracking accuracy by affecting the local magnetic fields. Moreover, the need for wires in these systems can interfere with the use of laparoscopic instruments.
  • Combined methods can also be used, for non-limiting example, a combination of at least one passive optical marker and at least one EM sensor on a tool in order to minimize the effects of occasional blocking of the line-of-sight of the optical markers and distortion in the EM system.
  • at least one force/torque sensor can be mounted on at least one surgical object. This exemplary combination can accurately measure position, orientation, velocity, acceleration, motion smoothness, and force applied by the tool.
  • the inertial sensor typically a gyroscope, accelerometer, velocity sensor and any combination thereof, measures the angular velocity and the acceleration of the target. These motion parameters are used to compute the position and orientation of the tracked object.
  • the main advantage of this device is the high update rate, but it suffers from error accumulation over time, resulting in larger errors in velocity and position, which, in some clinical applications, can not be tolerated. Rectification algorithms can be applied in order to reduce the effects of error accumulation.
  • the system can determine which procedure is being executed and the critical points of the procedure.
  • the procedure is input to the system by an operator.
  • Ultrasound can be used in much the same manner as optical tracking.
  • three or more emitters are mounted on a surgical object. Each emitter generates a sonic signal that is detected by a receiver placed at a fixed known position in the environment. Based on the sonic signals generated by the emitters, the system can determine their positions by triangulation. Combining three receivers, the tracker can estimate also the orientation of the target.
  • these systems suffer from the environment-dependent velocity of the sound waves, which varies with temperature, pressure and humidity. The loss of energy of the ultrasonic signal with distance also limits the range of tracking.
  • acoustic tracking requires line-of- sight, lack of which can affect the quality of the signal.
  • a restricting mechanism which can restrict movement of an endoscope, at least one said surgical tool, and any combination thereof.
  • the restricting mechanism can be, for non-limiting example, a brake, a stop, a catch, software configured to restrict movement, any other means of restricting movement known in the art, and any combination thereof.

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Public Health (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Surgery (AREA)
  • Primary Health Care (AREA)
  • Epidemiology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Veterinary Medicine (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Business, Economics & Management (AREA)
  • Urology & Nephrology (AREA)
  • Biophysics (AREA)
  • Human Computer Interaction (AREA)
  • Pathology (AREA)
  • Gynecology & Obstetrics (AREA)
  • Business, Economics & Management (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Robotics (AREA)
  • General Physics & Mathematics (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)

Abstract

The present invention provides a system for identifying at least one critical point in a procedure, comprising: a. at least one imaging device configured to provide, for any given time t, at least one first image in a field of view of a surgical environment and, for at least one time t + ∆t, to provide at least one second image in said field of view; b. processor in communication with said at least one imaging device; and, c. communicable database configured to (i) store said at least one first parameter at said time t; and (i) store said at least one first parameter at said time t + ∆t; wherein, from comparison between said at least one first parameter at said time t and said at least one first parameter at said time t + ∆t, said at least one critical point is identifiable.

Description

AUTONOMIC SYSTEM FOR DETERMINING CRITICAL POINTS DURING
LAPAROSCOPIC SURGERY
FIELD OF THE INVENTION
The present invention generally pertains to a system and method for providing a means and method for determining critical points in a procedure.
BACKGROUND OF THE INVENTION
In systems where one or more surgical tools is automatically or autonomically controlled, automatic detection of critical points in a procedure is needed in order to determine whether, for example, a procedure is complete, or whether an deviation has occurred in a procedure. Automatic detection of critical points is also needed in order to identify surgical tasks in a surgical procedure, for example, in removal of a tumor from an organ, typical surgical tasks can include incising tissue around the tumor, grasping and removing the tumor, suction to remove blood and keep the surgical environment clear, cauterizing tissue, and suturing the incision. Off-line identification of critical points such as the beginnings and ends of the surgical tasks can vastly improve efficiency in creating databases of surgical tasks and procedures which can be used for, among other things, training and assessing of surgical personnel and improving the quality of the procedures that are taught and used.
Furthermore, in laparoscopic surgery, automatic detection is complicated by the fact that, if the display of the surgical environment (usually 2 dimensional) is used for detection of the action of the tool or tools, then detection must be made from a view restricted to a two- dimensional view.
It is therefore a long felt need to provide a system for laparoscopic surgery which does not require manual input to determine critical points in a laparoscopic procedure.
SUMMARY OF THE INVENTION
It is an object of the present invention to disclose a system and method for providing a means and method for determining critical points in a procedure.
It is another object of the present invention to disclose a system for identifying at least one critical point in a procedure, comprising: a. at least one imaging device configured to provide, for any given time t, at least one first image in a field of view of a surgical environment and, for at least one time t + At, to provide at least one second image in said field of view; b. at least one processor in communication with said at least one imaging device; said at least one processor is configured to (z) analyze said at least one first image; (z'z) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (Hi) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t, (z'v) identify from at least one second image said at least one item, and, (v) calculate from said least one second image, for said at least one item, said at least one first parameter at said time t + At; and, c. at least one communicable database configured to (z) store said at least one first parameter at said time t; and (z'z) store said at least one first parameter at said time t + At; wherein, from comparison between said at least one first parameter at said time t and said at least one first parameter at said time t + At, said at least one critical point is identifiable.
It is another object of the present invention to disclose the system as described above, wherein said database comprises at least one prestored parameter for any given time t.
It is another object of the present invention to disclose the system as described above, wherein said at least one processor is configured to (z) compare said at least one first parameter at said time t to said at least one prestored parameter at said time t; and (z'z) identify, from said comparison, at least one said critical point.
It is another object of the present invention to disclose the system as described above, wherein, upon detection of a difference between said at least one first parameter and said at least one prestored parameter less than a predetermined value, a critical point is determinable.
It is another object of the present invention to disclose the system as described above, wherein said predetermined value is in a range of about 0.1% to about 15%.
It is another object of the present invention to disclose the system as described above, wherein said predetermined value is about 5%. It is another object of the present invention to disclose the system as described above, wherein said prestored parameter is calculable from a third image for said time ί in a procedure selected from a group consisting of: said procedure executed at a different time, said procedure executed on a simulator; said procedure simulated in a processor, and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said surgical procedure is selected from a group consisting of an identifiable unit, a surgical task, a complete procedure, and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein, upon detection of a difference between said at least one first parameter at said time t and said at least one first parameter at said time t + At is greater than a predetermined value, a critical point is determinable.
It is another object of the present invention to disclose the system as described above, wherein said predetermined value is in a range of about 0.1% to about 15%.
It is another object of the present invention to disclose the system as described above, wherein said predetermined value is about 5%
It is another object of the present invention to disclose the system as described above, wherein said comparison is a real-time comparison.
It is another object of the present invention to disclose the system as described above, wherein said at least one critical point is selectable from a group consisting of: a location in said surgical environment, a beginning of a procedure, an end of a procedure, an intermediate point in a procedure and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein a plurality of said locations in said surgical environment are determinable from at least one said image.
It is another object of the present invention to disclose the system as described above, wherein a path is determinable from said plurality of said locations.
It is another object of the present invention to disclose the system as described above, wherein a member of an overlay group consisting of: said path, at least one said critical point, and any combination thereof is overlayable on at least one said image in said field of view.
It is another object of the present invention to disclose the system as described above, wherein at least one member of said overlay group is storable in conjunction with said at least one said image.
It is another object of the present invention to disclose the system as described above, wherein at least one indication is displayable upon determination of said at least one critical point.
It is another object of the present invention to disclose the system as described above, wherein said at least one indication is selected from a group consisting of a restricting mechanism, a visible signal, an audible signal, a tactile signal and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said restricting mechanism is configured to restrict movement of a member of a group consisting of: at least one endoscope, at least one said surgical object and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said visible signal is selected from a group consisting of a constant-color pattern, a varying-color pattern, a constant- shape pattern, a varying-shape pattern, constant-size pattern, a varying-size pattern, an arrow, a word and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said audible signal is selected from a group consisting of a constant-pitch sound, a varying-pitch sound, a constant-loudness sound, a varying-loudness sound, a word and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said tactile signal is selected from a group consisting of a vibration, a constant- pressure signal, a varying-pressure signal, a stationary signal, a moving signal and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said tactile signal is applyable to a member of a group consisting of: a head, a neck, a torso, an arm, a wrist, a hand, a finger, a leg, an ankle, a toe and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said critical point for said at least one item is ascertainable from at least one second parameter derivable from a member of a group consisting of: a signal from a sensor, a forward kinematics calculation, an inverse kinematics calculation, a CT image, an MRI image, an X-ray image and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein communication between said at least one sensor and said at least one item is selected from a group consisting of: mechanical communication, wired communication, wireless communication and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said sensor is selected from a group consisting of an electromagnetic sensor; an ultrasound sensor; an inertial sensor to sense the angular velocity and the acceleration of the tool or other item; an accelerometer, a motion sensor, an EVIU, a sensor wearable by an operator, a sensor attachable to an item, an RFID tag attachable to an item, an ultrasound sensor, an infrared sensor, gyro-meter, tachometer, shaft encoder, rotary encoder, strain gauge and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein a member of a group consisting of said at least one first parameter, said at least one prestored parameter, said at least one second parameter, and any combination thereof is selected from a group consisting of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, state of an item, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein a member of a group consisting of said at least one first parameter, said at least one prestored parameter, said at least one second parameter, and any combination thereof is a function of time.
It is another object of the present invention to disclose the system as described above, wherein said at least one critical point is identifiable in a manner selected from a group consisting of: a comparison of a plurality of consecutive first parameters to a plurality of consecutive prestored parameters, a comparison of a plurality of consecutive second parameters to a plurality of consecutive prestored parameters, and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said at least one imaging device is configured to real time provide a plurality of said images.
It is another object of the present invention to disclose the system as described above, wherein said database is configured to store at least one record of at least one procedure, each said at least one record comprising a member of a group consisting of: said at least one image, said at least one first parameter, said at least one prestored parameter, said at least one second parameter, said at least one critical point, a fixed point, an object in said surgical environment, an angle, a distance, a reference grid, a horizon, an area, a volume, and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said at least one record is selectable based upon said identifier.
It is another object of the present invention to disclose the system as described above, wherein said identifier is selected from a group consisting of: an identifier of an operator, an identifier of an operating room, a physical characteristic of an operating room, a start time of a procedure, end time of a procedure, duration of a procedure, date of a procedure, an identifier of a patient, a physical characteristic of a patient, an outcome of a procedure, length of hospital stay for a patient, a readmission for a patient, and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said physical characteristic of said operating room during said procedure is selected from a group consisting of: temperature, humidity, time of cleaning, date of cleaning, cleaning procedure, cleaning material, type of lighting and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said physical characteristic of said patient is selected from a group consisting of: age, height, weight, body mass index, health status, medical status, and any combination thereof. It is another object of the present invention to disclose the system as described above, wherein said outcome of said procedure is selected from a group consisting of: a successful aspect, a partially successful aspect, a partial failure in an aspect, a complete failure in an aspect, and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein said at least one image is selected from a group consisting of: a 2D image, a 3D image, a panoramic image, a high resolution image, a 3D image reconstructed from at least one 2D image, a 3D stereo image and any combination thereof.
It is another object of the present invention to disclose the system as described above, wherein at least one of a group consisting of: said at least one critical point, a fixed point, a path, a suggestion, an instruction, a distance, an angle, an area, a volume, a size scale, information on a medical history of a patient, and any combination thereof is overlayable on said at least one image.
It is another object of the present invention to disclose a system for identifying at least one critical point in a procedure, comprising: a. at least one imaging device configured to provide, for any given time t, at least one first image in a field of view of a surgical environment; b. at least one processor in communication with said at least one imaging device; said at least one processor is configured to (z) analyze said at least one first image; (ii) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (z'z'z) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t, (iv) compare said at least one first parameter at said time t to at least one prestored parameter at said time t, and, c. at least one communicable database configured to (z) store said at least one first parameter at said time t; and (ii) store said at least one prestored parameter for any given time t; wherein, from comparison between said at least one first parameter at said time t and said at least one prestored parameter at said time t, said at least one critical point is identifiable.
It is another object of the present invention to disclose the system as described above, wherein said at least one processor is configured to (z) provide, for at least one time t + At, at least one second image in said field of view; (ii) identify from at least one second image said at least one item; (Hi) calculate from said least one second image said at least one first parameter at said time t + At; (iv) store said at least one first parameter at said time t + At; and (v) identify, from said comparison, at least one said critical point.
It is another object of the present invention to disclose a method for identifying at least one critical point in a procedure, comprising steps of: a. providing a system for identifying at least one critical point in a procedure, comprising: i. at least one imaging device configured to provide, for any given time t, at least one first image in a field of view of a surgical environment and, for at least one time t + At, to provide at least one second image in said field of view; ii. at least one processor in communication with said at least one imaging device; said at least one processor is configured to (i) analyze said at least one first image, (ii) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (Hi) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t; (iv) identify from at least one second image said at least one item, and (v) calculate from said least one second image, for said at least one item, said at least one first parameter at said time t + At; and, iii. at least one communicable database configured to (i) store said at least one first parameter at said time t; and; and (ii) store said at least one first parameter at said time t + At; b. providing, via said at least one imaging device, said at least one first image at said time t and said at least one second image at said time t + At; c. analyzing said at least one first image and determining, from said at least one first image, said at least one item; d. calculating from said at least one first image, for said at least one item, said at least one first parameter at said time t; e. storing said at least one first parameter at said time t; f. analyzing said at least one second image and determining, from said at least one second image, said at least one item; g. calculating from said at least one second image, for said at least one item, said at least one first parameter at said time t + At; h. storing said at least one first parameter at said time t + At; and i. comparing said at least one first parameter at said time t to said at least one first parameter at said time t + At, thereby, from said comparison, identifying said at least one critical point.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of storing in said database at least one prestored parameter for any given time t.
It is another object of the present invention to disclose the method as described above, additionally comprising steps of (z) comparing said at least one first parameter at said time t to said at least one prestored parameter at said time t; and (ii) identifying, from said comparison, at least one said critical point.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of determining a critical point by detecting a difference between said at least one first parameter and said at least one prestored parameter less than a predetermined value.
It is another object of the present invention to disclose the method as described above, additionally comprising a steps of selecting said predetermined value to be in a range of about 0.1% to about 15%.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said predetermined value to be about 5%.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of calculating said prestored parameter from a second image for said time t in a procedure selected from a group consisting of: said procedure executed at a different time, said procedure executed on a simulator; said procedure simulated in a processor, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said surgical procedure is selected from a group consisting of an identifiable unit, a surgical task, a complete procedure, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of determining a critical point by detecting a difference between said at least one first parameter at time t and said at least one first parameter at time t + At greater than a predetermined value.
It is another object of the present invention to disclose the method as described above, additionally comprising steps of selecting said predetermined value to be in a range of about 0.1% to about 15%.
It is another object of the present invention to disclose the method as described above, additionally comprising steps of selecting said predetermined value to be about 5%.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of real-time performing said comparison.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said at least one critical point from a group consisting of: a location in said surgical environment, a beginning of a procedure, an end of a procedure, an intermediate point in a procedure and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of determining a plurality of said locations in said surgical environment from at least one said image.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of determining a path from said plurality of said locations in said surgical environment.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of overlaying on at least one said image in said field of view a member of an overlay group consisting of: said path, at least one said critical point, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of storing at least one member of said overlay group in conjunction with said at least one said image. It is another object of the present invention to disclose the method as described above, additionally comprising a step of displaying at least one indication upon determination of said at least one critical point.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said indication from a group consisting of: a restricting mechanism, a visible signal, an audible signal, a tactile signal and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of restricting, by means of said restricting mechanism, movement of a member of a group consisting of: at least one endoscope, at least one said item and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said visible signal from a group consisting of a constant-color pattern, a varying-color pattern, a constant- shape pattern, a varying-shape pattern, constant-size pattern, a varying-size pattern, an arrow, a word and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said audible signal from a group consisting of a constant-pitch sound, a varying-pitch sound, a constant-loudness sound, a varying-loudness sound, a word and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said tactile signal from a group consisting of a vibration, a constant-pressure signal, a varying-pressure signal, a stationary signal, a moving signal and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of applying said tactile signal to a member of a group consisting of: a head, a neck, a torso, an arm, a wrist, a hand, a finger, a leg, an ankle, a toe and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of ascertaining said critical point for said at least one item from at least one second parameter derivable from a member of a group consisting of: a signal from a sensor, a forward kinematics calculation, an inverse kinematics calculation, a CT image, an MRI image, an X-ray image, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of providing communication between said sensor and said at least one item selected from a group consisting of: in mechanical communication, wired communication, wireless communication and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said sensor from a group consisting of an electromagnetic sensor; an ultrasound sensor; an inertial sensor to sense the angular velocity and the acceleration of the tool or other item; an accelerometer, a motion sensor, an IMU, a sensor wearable by an operator, a sensor attachable to an item, an RFID tag attachable to a surgical object, an ultrasound sensor, an infrared sensor, gyro-meter, tachometer, shaft encoder, rotary encoder, strain gauge and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said at least one first parameter from a group consisting of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, state of an item, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said at least one prestored parameter from a group consisting of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, state of an item, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of said at least one second parameter from a group consisting of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, state of an item, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of providing a member of a group consisting of said at least one first parameter, said at least one prestored parameter, said at least one second parameter, and any combination thereof as function of time.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of identifying said at least one critical point in a manner selected from a group consisting of: comparing a plurality of consecutive first parameters to a plurality of consecutive prestored parameters, comparing a plurality of consecutive first parameters to a plurality of consecutive second parameters, comparing a plurality of consecutive second parameters to a plurality of consecutive prestored parameters and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of providing, by means of said at least one imaging device, a plurality of said images.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of storing in said database at least one record of at least one procedure, each said at least one record comprising a member of a group consisting of: said at least one image, said at least one first parameter, said at least one prestored parameter, said at least one second parameter, said at least one critical point, a fixed point, an object in said surgical environment, an angle, a distance, a reference grid, a horizon, an area, a volume, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said at least one record based upon said identifier.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said identifier from a group consisting of: an identifier of an operator, an identifier of an operating room, a physical characteristics of an operating room, a start time of a procedure, end time of a procedure, duration of a procedure, date of a procedure, an identifier of a patient, a physical characteristic of a patient, an outcome of a procedure, length of hospital stay for a patient, a readmission for a patient, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said physical characteristic of said operating room from a group consisting of: temperature, humidity, time of cleaning, date of cleaning, cleaning procedure, cleaning material, type of lighting and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said physical characteristic of said patient from a group consisting of: age, height, weight, body mass index, health status, medical status, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said outcome of said procedure from a group consisting of: a successful aspect, a partially successful aspect, a partial failure in an aspect, a complete failure in an aspect, and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of selecting said at least one image from a group consisting of: a 2D image, a 3D image, a panoramic image, a high resolution image, a 3D image reconstructed from at least one 2D image, a 3D stereo image and any combination thereof.
It is another object of the present invention to disclose the method as described above, additionally comprising a step of overlaying on said at least one image at least one of a group consisting of: said critical point, a fixed point, a path, a suggestion, an instruction, a distance, an angle, an area, a volume, a size scale, information on a medical history of a patient, and any combination thereof.
It is another object of the present invention to disclose a method for identifying at least one critical point in a procedure, comprising steps of: a. providing a system for identifying at least one critical point in a procedure, comprising: i. at least one imaging device configured to provide, for any given time t, at least one first image in a field of view of a surgical environment; ii. at least one processor in communication with said i at least one imaging device; said at least one processor is configured to (i) analyze said at least one first image; (ii) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (Hi) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t, (iv) compare said at least one first parameter at said time t to at least one prestored parameter at said time t, and, iii. at least one communicable database configured to (i) store said at least one first parameter at said time t; and (ii) store said at least one prestored parameter for any given time t; b. providing, via said at least one imaging device, said at least one image of a field of view at said time t; c. analyzing said at least one image and determining, from said at least one image, said at least one item; d. calculating from said at least one image, for said at least one item, said at least one first parameter at said time t; e. storing said at least one first parameter at said time t; f. analyzing said at least one second image and determining, from said at least one second image, said at least one item; and g. comparing said at least one first parameter at said time t to said at least one prestored parameter at said time t, thereby, from said comparison, identifying said at least one critical point.
It is another object of the present invention to disclose the method as described above, additionally comprising steps of: (i) analyzing said at least one second image and determining, from said at least one second image, said at least one item; (ii) calculating from said at least one second image said at least one first parameter at said time t + At of said at least one item; (Hi) storing said at least one first parameter at said time t + At; (iv) comparing said at least one first parameter at said time t to said at least one first parameter at said time t + At; and, (v) identifying, from said comparison, said at least one critical point.
BRIEF DESCRIPTION OF THE FIGURES
In order to better understand the invention and its implementation in practice, a plurality of embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, wherein
Fig. 1 schematically illustrates an image of a surgical environment; Fig. 2 schematically illustrates movement of a tip of a surgical tool;
Fig. 3A schematically illustrates speed of the tool tip and Fig. 3B schematically illustrates acceleration of the tool tip during the procedure; Fig. 4A schematically illustrates speed of the tool tip, Fig. 4B schematically illustrates acceleration of the tool tip and Fig. 4C schematically illustrates jerk of the tool tip during the first part of the procedure;
Fig. 5 schematically illustrates critical points for a path;
Fig. 6 schematically illustrates starting and ending point for a path;
Fig. 7 schematically illustrates an ending point for a procedure;
Fig. 8 schematically illustrates display overlay indicating an end-of-procedure;
Figs. 9 and 10 schematically illustrate an embodiment of a method of automatically assessing or automatically training an operator; and
Fig. 11A-B depicts a robot arm including IMUs.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
The following description is provided, alongside all chapters of the present invention, so as to enable any person skilled in the art to make use of said invention and sets forth the best modes contemplated by the inventor of carrying out this invention. Various modifications, however, will remain apparent to those skilled in the art, since the generic principles of the present invention have been defined specifically to provide a means and method for determining critical points in a procedure.
The term "fixed point" hereinafter refers to a location in a surgical field which is fixed relative to a known location. The known location can be, for non-limiting example, an insertion point, a known location in a patient, a known location in an environment around a patient (e.g., an attachment point of a robotic manipulator to an operating table, a hospital bed, a wall of a room), a known location in a manipulation system, a known location in a practice dummy, or a known location in or on an operator. A fixed point can also be a point marked in an image in a field of view of a surgical environment.
The term "item" hereinafter refers to any identifiable thing within a field of view of an imaging device. An item can be something belonging to a body or something introduced into the body. Items also comprise things such as, for non-limiting example, shrapnel or parasites and non-physical things such as fixed points. The term "object" hereinafter refers to an item naturally found within a body cavity. Non- limiting examples of an object include a blood vessel, an organ, a nerve, and a ligament, as well as an abnormality such as a lesion and a tumor.
The term "tool" or "surgical tool" hereinafter refers to an item mechanically introducible into a body cavity. Non-limiting examples of a tool include a laparoscope, a light, a suction device, a grasper, a suture material, a needle, and a swab.
The term "surgical object" hereinafter refers to a surgical tool, a robotic manipulator or other maneuvering system configured to manipulate a surgical tool, at least a portion of a light source, and at least a portion of an ablator.
The term "operator" hereinafter refers to any of: a principal operator such as, but not limited to, the surgeon carrying out the main parts of the procedure, an assistant such as, but not limited to, a nurse and an observer such as, but not limited to, a senior surgeon providing instruction to or assessing a principal operator. An identifier for an operator can include, but is not limited to, a name, an ID number, a function and any combination thereof.
The term "identifiable unit" hereinafter refers to an identifiable purposive activity during a surgical operation, typically a minimal identifiable activity. Examples include, but are not limited to, movement of a needle and forceps to the site where a suture is to be made, making a knot in suture thread, activating fluid flow, and making an incision.
The term "surgical task" hereinafter refers to a connected series of at least one identifiable unit which comprises an identifiable activity. Non-limiting examples of surgical tasks that comprise more than one identifiable unit include, but are not limited to, making one suture, removing incised tissue from a surgical field, and clearing debris from a surgical field. A non-limiting example of a surgical task that comprises a single identifiable unit is making an incision.
The term "complete procedure" hereinafter refers to a connected series of at least one surgical task which forms an independent unit. For non-limiting example, closing an incision with one or more sutures will be referred to as a complete procedure.
The term "procedure" or "surgical procedure" hereinafter refers to at least a portion of a surgical operation, with the portion of the surgical operation including at least one identifiable unit. For non-limiting example, in increasing order of complexity, a procedure can comprise tying the knot in a suture, making a single suture, or closing an incision with a series of sutures. The term "automatic procedure" hereinafter refers to a procedure in which one surgical tool automatically responds to a movement or other action of a second surgical tool. Non-limiting examples of an automatically-executed procedure includes tracking a surgical tool by an endoscope or changing a lighting level in response to an increase in a perceived amount of smoke.
The term "autonomic procedure" hereinafter refers to a procedure which can be executed independently of actions of a surgeon or of other tools. Non-limiting examples of autonomic procedures include executing a complete suture and executing a plurality of sutures to close an incision.
The term "operator" hereinafter refers to any of: a principal operator such as, but not limited to, the surgeon carrying out the main parts of the procedure, an assistant such as, but not limited to, a nurse and an observer such as, but not limited to, a senior surgeon providing instruction to or assessing a principal operator. An identifier for an operator can include, but is not limited to, a name, an ID number, a function and any combination thereof.
The terms "aspect" or "aspect of an outcome" hereinafter refer to a part of the outcome of a procedure, where the outcome is the overall result of the procedure. Examples of aspects of an outcome include: amount of bleeding, speed of healing, return of an abnormality, occurrence of adhesions and patient discomfort.
The system of the present invention can determine critical points in a surgical procedure, either when the procedure is autonomously executed (without input from an operator) or when it is automatically executed (as an adjunct to an operator-executed procedure, such as moving an endoscope to ensure a continuously good view of suturing). The system preferably analyzes at least one image of at least a portion of a field of view of the surgical environment, and by means of said analysis, determines a critical point, a characteristic movement, a characteristic position, a characteristic orientation and any combination thereof. A plurality of critical points can comprise, for non-limiting example, a path for a surgical object to follow, an indication that a procedure has started (start of procedure), an indication that a procedure is complete (end of procedure) and any combination thereof.
From analysis of the critical points, for non-limiting example, at least one of the following can be determined:
• Between at least two critical points, a preferred path can be found. o The path can mark, for non-limiting example, movements of at least one surgical object during at least one procedure. For non-limiting example, paths can mark movement of a needle and movement of a forceps during creation of a suture.
o The path can mark the beginning and end of a line of sutures. Critical points can mark, for non-limiting example, the locations of the sutures. For non- limiting example, the path can be along the line of an incision, and would have to take into account the contours of the organ being sutured, possible movement of the organ, and ensuring that the suturing tool or tools can bypass any obstacles in the path.
• A critical point for start of a procedure can be determined by comparing the movement of at least one surgical object to stored movements, by determining that the location of at least one surgical object matches the location of at least one fixed point and any combination thereof. If the movements of at least one surgical object match movements characterizing a start of a procedure, the procedure is deemed to be starting.
• A critical point for an intermediate point in a procedure can be determined by comparing the movement of at least one surgical object to stored movements, by determining that the location of at least one surgical object matches the location of at least one fixed point and any combination thereof. If the movements of at least one surgical object match movements characterizing an intermediate point in a procedure, then an intermediate critical point has occurred. For non-limiting example, in making a single suture, an intermediate critical point can occur where one end of a suture thread passes under the other end and the operator must release a needle from a forceps and grasp it again.
• A critical point for completion of a procedure can be determined by comparing the movement of at least one surgical object to stored movements, by determining that the location of at least one surgical object matches the location of at least one fixed point and any combination thereof. If the movements of at least one surgical object match movements characterizing an end of a procedure, the procedure is deemed complete.
In preferred embodiments, in addition to identifying critical points, the system can execute at least one of the following: • A stored history can be tagged with at least one identifier, to enhance and simplify searching at least one library comprising at least one stored procedure. For non-limiting example, a procedure can be tagged with an identifier of at least one operator, with a type of at least one procedure carried out, and with at least one characteristic of a patient. For non- limiting example, this could be used to determine the quality of outcome for appendectomies performed by Dr. Jones.
• The system can interface with at least one surgical object, such as, but not limited to, a robotic maneuvering system, a surgical tool, a suction device, lighting, an ablator, a fluid supply and any combination thereof. For non-limiting example, if the system determines that there is effusion of blood from an incision, it could command that a suction device be brought into the region of blood effusion and that suction be applied to the blood.
• At least one fixed point can be marked so that the system can later be directed to move a surgical object to a selected fixed point, to move a surgical object to a known orientation, to set the surgical object into a known state, and any combination thereof. The state of the surgical object can include, but is not limited to, a lighting level, a fluid flow level, a state of activated/deactivated, a known zoom, and a known temperature.
Other uses of a fixed point include finding a preferred path between two fixed points. The path need not be a straight line. In some embodiments, it can bypass obstacles and, in some embodiments, can take into account non-flatness of, for example, an organ, if the path is to follow the surface of an organ. In some embodiments, a path can take into account movement such as, for non-limiting example, movement of an organ. In some embodiments, the path can take into account a desired distance between two tools at the end of a path, for non-limiting example, if the path is that of a retractor separating tissue after an incision.
In some embodiments, an artificial intelligence (AI) system analyzes at least one scene in at least one field of view and, from the analysis (and possibly from other provided information) forms an understanding of what is occurring. From this understanding, the AI system can identify at least one critical point and, preferably, can respond appropriately.
From analysis of at least one image, the system can calculate at least one performance metric and at least one parameter, and can derive at least one of an identifiable unit and a critical point. Image-based tracking can identify the properties of at least one surgical object, including a surgical tool attached to a robotic arm, a surgical tool controlled directly by an operator, a static surgical object in a surgical environment and a surgical object connectable to a surgical object in the surgical field, as described herein. For non-limiting example, this information can be used to avoid an obstacle, to instruct an operator to avoid an obstacle, to focus (for example, an endoscope) on a point of interest, to instruct an operator to focus on a point of interest, and any combination thereof. In some embodiments, the system can evaluate how an operator interacts with different objects to ascertain the operator's intent or identify the task that is currently being performed.
A non-limiting example of a plurality of parameters comprising an identifiable unit is given by the knot-tying identifiable unit, which can involve parameters such as total time spent tying a knot, idle time during knot tying, search time, approach time taken to reach the site of the knot, speed of the tools during knot tying, motion smoothness, bimanual dexterity, the length of the path followed by each tool, and the distance efficiency, which is a comparison of the actual path lengths with an optimal path length.
Both a parameter and an identifiable unit can be a function of time; either or both can change during a procedure. Preferably, any parameter or identifiable unit which can be a function of time is stored as a function of time.
The movements of at least one surgical tool held by an operator or moved by the system and the forces exerted by (and on) this surgical tool can be determined, from analysis of at least one image of at least a portion of the surgical environment, by a tracking subsystem, and any combination thereof. Image analysis can be used to determine the location of at least one patient feature such as, but not limited to, an organ, a blood vessel, a nerve, a lesion, a tumor, a tissue, a bone, a ligament and any combination thereof. Surgical tool identification and tracking and image analysis can be used to determine the location of at least one non-moving surgical tools such as a swab and also to determine a location of non-surgical items in the surgical environment. Non-limiting examples of a non-surgical item in a surgical environment include such things as a glass shard or a bomb fragment.
Fig. 1 illustrates an image of a portion of a surgical environment (100). Both tools (110) and organs (120) are identifiable in the image.
The totality of the data on location, orientation and movements of surgical objects provides spatiotemporal 3 -dimensional data which characterize the surgical environment and the at least one item within it. The spatiotemporal 3 -dimensional data can be analyzed in real time, stored in a database for later analysis and any combination thereof. Other measureable and storable data include: grasping force, torsion about the tool axis, Cartesian forces, and any combination of these and the positions, orientations and movements disclosed above. In some embodiments, the system also can store at least one image of at least a portion of at least one procedure; this can also be synchronized with the force and position data.
Yet more measureable and storable data include: a distance between two items, a distance between an item and a fixed point, a distance between two fixed points, an angle between two items, an angle between an item and a reference line, an angle between two reference lines, an angle between at least one fixed point and at least one item, an angle between a reference line and at least one item, a volume between items, a volume between reference lines, a volume between at least one item and a reference line, a volume between at least one item and a fixed point, an angle between two items, and any combination thereof.
In some embodiments where the motion of at least one tool is tracked, Table I gives a non- limiting example of metrics which can be assessed. In a given embodiment, any combination of metrics can be used.
Table I Motion-based Metrics
Parameter (Metric) Description
Time related
Time Total time taken to perform the task
Idle time Percentage of time where a surgical object is
considered to be non-moving
Approach time Time taken to reach the target point
Speed Rate of change of a surgical object's position
Maximum speed
Speed profile Shape of the speed curve
Acceleration Rate of change of a surgical object's speed
Motion smoothness Motion analysis parameter based on the third time- derivative of position, which represents a change in acceleration
Search time Percentage of time spent in the "search zone"
Space-related
Path length Length of the curve described by the tip of a surgical
object while performing the task
Distance efficiency Relationship between measured path length and
shortest path. Describes the economy of movements
Depth perception Total distance travelled by a surgical object along its
axis
Transit profile 2D transit trajectory projected onto a plane Deviation on horizontal Deviation from the ideal course in the horizontal and plane and vertical plane vertical directions
Response orientation Characterizes the amount of rotation about the axis of
a surgical object
Economy of area (EOA) Relationship between the maximum area occupied by
a surgical object and the total path length
Economy of volume (EOV) Relationship between the maximum volume occupied
by a surgical object and the total path length
Number of movements Number of movements made to complete a task
(measured as number of zero crossings on the acceleration/time profile)
Force-related
Force range Difference between the minimum and the maximum
force applied during a task
Interquartile force range Selects the 50 % of the data closest to the median so
that outliers do not have an effect on the overall
metric.
Integral of the force Provides a measure of high forces and the amount of
time that forces are high.
Integral of the grasping Provides a measure of high forces and the amount of force time that forces are high.
Integral of the Cartesian Provides a measure of high forces and the amount of force time that forces are high.
First derivative of the force Indicates consistency of force application.
Second derivative of the Indicates consistency of force application.
force
Third derivative of the force Indicates smoothness of force application
Fig. 2 shows, schematically, the 3D movements, over time, of the tip of a surgical tool during a procedure. Fig. 3A shows the speed of the tool tip during the procedure, while Fig. 3B shows the acceleration of the tool tip during the procedure. The speed, acceleration and jerk for the first part of the procedure are shown in Figs. 4A, B and C, respectively. From these, the metrics of Table II can be calculated.
Table II shows exemplary means of calculating the metrics of Table I.
force d F metric d FmetriC ~ jiF^ I dt3 dt
In Table II, to is the start time of the procedure, if is the end time, the motion has amplitude R and vector (rx, ry, rz), where R = jr} + Ty + rz 2, 5/^ and Sright are, respectively, the time spent using the left hand and the time spent using the right hand, and /is a measured force.
The system of the present invention can comprise an advanced artificial intelligence (AI) system running on at least one processor which is capable of analyzing a scene in a field of view (FOV), as captured in real time by an imaging device and, from the analysis (and, in some embodiments, additional information from other sensors) forming an understanding of what is occurring. From this understanding, the system can derive at least one parameter, such as one of the metrics disclosed herein, and, from one or more parameters, determine at least one critical point in the surgical environment.
Although capture of the at least one image of a field of view is carried out in real time and storage of at least one image is preferably carried out in real time, at least one of the other steps in the determination of a critical point can be carried out either in real time or off-line. The image captured by the imaging device can be a 2D image, a 3D image, a panoramic image, a high resolution image, a 3D image reconstructed from at least one 2D image, a 3D stereo image and any combination thereof.
A non-limiting example of a complete procedure comprising a number of surgical tasks, with each surgical task comprising an identifiable unit is suturing an incision, where the suturing process comprises a plurality of sutures. In this example, the process of suturing an incision comprises a procedure consisting of a number of surgical tasks. Movement of at least one tool to the site of a first suture could be treated as a first surgical task. Each suture could constitute a surgical task; each movement of a tool from a completed suture to a next suture could constitute a surgical task, and movement of a tool away from a last suture could constitute a final surgical task. Therefore, in this example, if an incision is closed with 4 sutures, the procedure would comprise 1 (move-to-first-suture-site) + 4 (suture) + 3 (move between sutures) + 1 (move away from last suture) = 9 surgical tasks. It would also comprise 8 critical points, namely 2 critical points for each suture. The first surgical task would be moving at least one tool to a first critical point, the second would move at least one tool from the first critical point through a set of suturing movements to a second critical point, the movement from the first suture to the second suture is movement from the second critical point to a third critical point, which is the first critical point of the second suture, and so on.
Similarly, a surgical tasks can be broken down into further surgical tasks or into identifiable units, and critical points assigned to each of the further surgical tasks or identifiable units. For non-limiting example, creating a single suture can comprise several identifiable units, including at least one of the following: (1) inserting a needle through the tissue, (2) pulling suturing thread through the tissue, (3) grasping one end of the suturing thread with a grasper, (4) cutting one end of the thread, (5) passing one end of the thread around the other to make a first tie, (6) pulling the first tie tight, (7), passing one end of the thread around the other to make a second tie, thus forming a knot, (8) pulling the knot tight, (7) and (8) clipping short the thread ends, and (9) removing the clipped ends. In this example, the surgical task of creating a suture would comprise a minimum of 18 critical points, namely, a critical point at the beginning and end of each identifiable unit.
Critical points can also be established at a point where there is a change in movement. For non-limiting example, movement of a tool from the location of a completed suture to the location of a next suture often involves a smooth movement upward and laterally away from both the completed suture and the tissue, followed by a downward, lateral movement to the location of the next suture. A critical point can be established at the beginning of the upward, lateral movement, at the end of the smooth movement, where the speed and direction of movement change, at the beginning of the downward, lateral movement and at the end of the downward, lateral movement, where the speed of movement again changes.
An idealized example of movement of a tool is shown in Fig. 5. In this example, the movement of the tool is shown by the solid line (550), and an idealized movement of the tool is shown by the dashed line (560). The dashed line is displaced downward slightly for clarity. Each critical point (580) in the actual movement (550) indicates where the slope of the curve (550) changes.
In some embodiments, the beginning of a procedure can be determined if at least one calculated first parameter is substantially equal to at least one prestored parameter. A beginning of a procedure can similarly be determined if at least one calculated first parameter at a time t + At is substantially different from the same at least one first parameter at time t.
In the example of Fig. 5, the beginning of the procedure occurs when the position, speed and acceleration of the tool (570) are substantially the same as the prestored parameters of position, speed and acceleration (575).
In preferred embodiments, two first parameters or a first parameter and a prestored parameter are deemed to be different if the difference between the two is greater than a predetermined amount. In some embodiments, the predetermined amount can be in the range of about 0.1% to about 15%. In a preferred embodiment, the predetermined amount is about 5%. The two first parameters or the first parameter and the prestored parameter are deemed to be substantially the same if the difference between the first parameter and the prestored parameter is no greater than the predetermined amount.
The exemplary tool movement shown in Fig. 5 can comprise a surgical task with a single beginning (570), a single end (590) and a number of critical points, or it can comprise up to 9 identifiable units, each comprising the minimum number, 2, of critical points.
A critical point can be established based on any of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
In some embodiments, the system identifies surgical tools in the working area; in preferred embodiments, objects such as organs, lesions, bleeding and other items related to the patient are identified, smoke, flowing fluid, and the quality of the lighting (level, dark spots, obscured spots, etc.). If smoke, flowing fluid or bleeding is identified, the system can respond by e.g., virtual smoke or fogging removal (removal of smoke of fogging from an image via software), increasing a lighting level, providing light from an additional direction or angle, starting smoke or fog removal measures such as flowing fluid across a lens or through an area obscured by smoke or fog, starting suction, alerting an operator to the bleeding, clarifying an image either in software or my changing zoom or focus, applying adaptive optic correction, and any combination thereof.
A basis of a determination of the parameters from which critical points can be identified can be a determination of at least one first parameter of at least one item in the field of view. This first parameter can be compared to either the first parameter at a different time or to a prestored parameter which is a second version of the same parameter, from a similar procedure executed at a different time. The prestored parameter, preferably stored in a database, can be an optimum or substantially optimum version of the first parameter, it can be a procedure executed by the same surgeon at a different time, a procedure executed by a different surgeon, a procedure executed on a simulator, a simulated procedure, and any combination thereof. The prestored parameter can also be an average. The average can be generated by averaging together a plurality of procedures and extracting a prestored parameter from the averaged procedure, or the average can be generated by extracting the parameter for each of the plurality of procedures and averaging the parameters.
First and prestored parameters can be functions of time and, if so, are preferably stored as functions of time.
The at least one first parameter can be a 2D position, for non-limiting example, in the plane of the field of view, of at least a portion of the at least one item; a 2D orientation, for non- limiting example, in the plane of the field of view, of at least a portion of the at least one item; a 3D position of at least a portion of the at least one item; a 3D orientation of at least a portion of the at least one item; a 2D projection of a 3D position of at least a portion of the at least one item, a velocity of at least a portion of the at least one item, an acceleration of at least a portion of the at least one item, an angle of at least a portion of the at least one item, altering the state of at least a portion of the at least one item, and any combination thereof.
The movement of the at least one item can be selected from a group consisting of a maneuver of the item carried out by a robotic manipulator connected to the item, a movement of part of the item, a change in state of the item, and any combination thereof. Non-limiting examples of movement of an item include displacing it, rotating it, zooming it, or, for an item with at least one bendable section, changing its articulation. Non-limiting examples of movements of part of the item are opening or closing a grasper or retractor, or operating a suturing mechanism. Non-limiting examples of a change in state of an item include: altering a lighting level, altering an amount of suction, altering an amount of fluid flow, altering a heating level in an ablator, altering an amount of defogging, or altering an amount of smoke removal.
In preferred embodiments, two first parameters at different times and a first parameter and a prestored parameter are different when the difference between them is greater than 5%; parameters, either two first parameters or a first and prestored parameter are substantially the same if the difference between them is no greater than 5%.
A critical point, including a critical point defining a beginning of a procedure or an end of a procedure can be determined if the difference between a first parameter at any given time t and the first parameter at a different time t + At is, greater than a predetermined value.
In some embodiments, a critical point, including a critical point defining a beginning of a procedure or an end of a procedure can be determined if the difference between a first parameter (that of the current procedure) and a prestored parameter (that of the stored procedure) is less than a predetermined value. In some embodiments, the critical point is determinable when the first parameter matches a prestored parameter tagged as a critical point. In some embodiments, a critical point, including a critical point defining a beginning of a procedure or an end of a procedure can be determined if the difference between a first parameter (that of the current procedure) and a prestored parameter (that of the stored procedure) is greater than a predetermined value.
At least the database and preferably both the tracking subsystem and the database are in communication with at least one processor configured to analyze the spatiotemporal 3- dimensional surgical database. The result of the analysis can be determining a path for movement of at least one tool, determining a path start point, determining a path end point, identifying a start of a procedure, identifying a start of a feature, identifying an end of a procedure, identifying an end of a feature, determining a critical point, and any combination thereof.
In preferred embodiments, at least one of a path, a critical point, the start of a procedure, and the end of a procedure can be provided as an indication to an operator. The indication to the operator can be a visible signal (e.g., on a display), an audible signal, a tactile signal and any combination thereof.
The visible signal can be: a list of the identified at least one surgical procedure, a label indicating the at least one surgical procedure in a list of surgical procedures, and any combination thereof. The label can be: a color, a pattern, an arrow, a line, a constant- brightness light, a variable-brightness light, a constant-color light, a variable-color light, a patterned light and any combination thereof.
The audible signal can be: a word, a constant-loudness sound, a variable-loudness sound, a constant-pitch sound, a variable-pitch sound, a patterned sound and any combination thereof.
The tactile signal can be a vibration, a constant-pressure signal, a varying-pressure signal, a stationary signal, a moving signal and any combination thereof. The tactile signal can be applied to the head, the neck, the torso, an arm, a wrist, a hand, a finger, a leg, an ankle, a toe and any combination thereof.
The state of a surgical tool includes general properties such as its position, orientation, speed, and acceleration. It can also include tool-specific properties, such as location or relative location of a movable part of a surgical tool, such as, for non-limiting example, the locations of the faces of a gripper relative to each other (e.g., whether a gripper is open or closed), a lighting level, an amount of suction, an amount of fluid flow, a heating level in an ablator, an amount of defogging, an amount of smoke removal. There are various mechanisms by which these properties can be determined. Some of these mechanisms are described herein.
In some embodiments, the system can identify at least one surgical tool in a surgical field, either by means of image recognition or by means of at least one tag associated with at least one surgical tool. A tag can comprise color-coding or another mechanical label, or electronic coding, such as, but not limited to, radiofrequency signals. A radiofrequency signal can be the same for different surgical tools or it can differ for at least one surgical tool. The system can recognize a labelled tool from its mechanical or radiofrequency coding, a tool can be identified by an operator, and any combination thereof.
In preferred embodiments, surgical tools comprise neither a marker nor a sensor, although at least one sensor can be used as part of at least one robotic tool-handling system. In such preferred embodiments, the system determines position and orientation of at least one item via analysis of at least one image, preferably an image provided by a laparoscopic camera, and, preferably, at least a portion of the image is displayed and is visible to an operator. The analysis of the image, for non-limiting example, can be based on the shape of an item. Item position and orientation can be determined, for example, via geometrical equations using the analysis of the projection of an item in the image plane or via Bayesian classifiers for detecting an item in the endoscopic image; the search for a surgical object can be further restricted by means of computing the projection of the surgical object's insertion point into the abdominal cavity.
Determination of item position and orientation in an image -based system can be rendered more difficult by ambiguous image structures, occlusions cause by blockage of the line of sight (e.g., by other items), blood, organs, smoke caused by electro-dissection, and any combination thereof. Particle filtering in the Hough space can be used to improve the tracking process in the presence of smoke, occlusions or motion blurring.
Path calculation is applicable to manual, automatic and autonomic procedures. In a manual procedure, a path can be overlaid on at least one image of a surgical environment, guiding an operator as to a path to be followed.
In an automatic procedure (where a robotic system is assisting an operator), path calculation can be advantageous in predicting at least one movement of an operator and thereby increasing efficiency in following the operator's movements. For non-limiting example, knowledge of an expected path can enable the system to configure a robotic arm's movement so as to prevent the arm from later being forced into an awkward or highly non-optimal orientation. In another non-limiting example, in a system where both an endoscope and a fluid source are automatically controlled, knowledge of a path can enable the fluid source and the endoscope to be maneuvered so that the fluid source remains in a good position for clearing the field of view and does not become "trapped" by the endoscope as both move from one ablation region to another. As in a manual procedure, path calculation can be used to suggest a path to an operator, for non-limiting example, if an actual path appears to the system to be highly non-optimal.
In an autonomic procedure, in addition to avoiding awkward or non-optimal positioning, knowledge of a path can ensure that no robotically controlled surgical tool comes in unwanted contact with another tool or with tissue.
Knowledge that a procedure has ended can enable at least one of the following: determination of at least one next procedure to be carried out; a display of at least one next procedure, enabling an operator to select the next procedure; and autonomic selection of a next procedure.
Fig. 6 shows an illustrative, non-limiting example of finding a path, for an incision in a liver (610). The starting point (640) and ending point (650) for an incision (620) are shown; the incision will divide the right lobe of the liver (610), while avoiding the falciform ligament (630). The calculated path for the incision (620) is shown by a dashed line, which approximately parallels the falciform ligament (630).
Fig. 7 shows an illustrative, non-limiting example of determining the end of a procedure from the movements of tools. The procedure is suturing to close an incision. The locations of the sutures are schematically illustrated by the arcs (730, 740, 750). The movement of the tip of the grasper holding the needle (710) and the movement of the tip of a second grasper (720) are shown. For each suture, after the suture is complete, the tools move diagonally upward away from the suture location, before moving diagonally downward to the location of the next suture. After completion of the suturing (dashed circle, 770), an end to the procedure is determinable; movement of the graspers is nearly horizontally (760), instead of a diagonal rise.
In some embodiments, a best practice or optimal procedure can be recorded from a procedure carried out by an experienced surgeon; it can be a compilation of fragments of recorded procedures, as carried out by several surgeons, thus combining the best parts a plurality of procedures to generate one best-practice procedure; or it can be a computer-generated procedure.
Fig. 8 shows an illustrative, non-limiting example of a displayed laparoscopic image (800) with an overlay (810) indicating the end of a procedure, in this case, the end of a suturing procedure.
Figs. 9-10 show an exemplary embodiment of a method of finding at least one critical point in a surgical environment. In the method (Fig. 9), the first step is to acquire at least one image of a field of view of an imaging device (910). The image is analyzed (920), as described herein, to identify, in 3D, the position, orientation and movement of at least one tool and preferably all of the tools in the field of view, and, in some embodiments, the relationship of the at least one tool to the surgical environment (i.e., the organs, blood vessels, nerves, etc. of the patient) and, in some embodiments, the relationship of at least one tool to at least one other tool in the surgical environment. In some embodiments, the force exerted by or on at least one tool is also acquired. From a member of a group including: position, orientation, movement, relationship and any combination thereof, at least one parameter of the tool is calculated (930). At least one of the parameters is then compared with a stored version of the same parameter (940) which, in some embodiments, is an optimum version of the parameter. If (950) the parameters are substantially the same, then no critical point has been found, and the method continues (circle 1) by acquiring (610) another image. If (950) the parameter(s) are not substantially the same, then a critical point has been found (960) and the method continues (circle 2) with Fig. 10.
As shown in Fig. 10, if (1010) a path is being identified, then (1020) a check is made whether all critical points in the path have been defined. If all critical points have been defined (1040), a path is calculated between the points. In preferred embodiments, the calculated path will avoid obstacles. After calculating at least one path, the routine ends. If not all critical points have been defined, an additional critical point is identified (1050), as a current location of at least a part of at least one tool, preferably a tool previously identified (950) as having at least one parameter different from the stored value of that parameter and the method continues (circle 1) by acquiring (910) another image.
If (1010) the end of a procedure is being determined, then (1030) a check is made whether the difference in the parameter is characteristic of an end of a procedure. For non-limiting example, a simultaneous linear movement of two graspers can signal the end of a knot-tying procedure. If the difference in parameter is not characteristic of the end of a procedure, then the method continues (circle 1) by acquiring (910) another image. In some embodiments, a deviating parameter can be displayed, to warn an operator that an error may have occurred. If the difference in parameter is characteristic of the end of a procedure, then the routine ends.
Preferably, the system of the present invention functions in real time or near real time. However, any function or operation disclosed herein can be carried out either real time or offline at a later time.
In some embodiments, a fixed point or critical point can be tagged manually. Manual tagging can be by an operator indicating, by word, by gesture, or by touching a touchscreen, that a given point, such as the current position of a tool, is to be tagged as a critical point or as a fixed point. For non-limiting example, automatic tagging can occur when a system identifies a point as a critical point or a fixed point. For non-limiting example, in semi-automatic tagging, the system automatically identifies a point as a possible critical point or fixed point and provides an indication of the possible existence of a critical point or fixed point. The operator then manually confirms (or denies) the existence of the critical point or fixed point.
Distance, area, angle and volume can be determined from members of a group consisting of fixed points, critical points and any combination thereof.
Two or more points can define at least one distance. Distance can be displayed as an individual, pair- wise distance, as a total distance, and any combination thereof. A marker can indicate a point, a mark indicating the extent of distance, and any combination thereof.
Three or more points can define at least one angle. Angle size can be displayed as an individual, trio-wise angle, as a total angle, and any combination thereof. A marker can indicate a point, an edge, a mark indicating the extent of an angle, and any combination thereof.
Three or more points can also define an area; multiple areas can be defined. A marker can indicate a boundary for an area, the size of an area, a selected area, a cumulative size for selected areas, and any combination thereof.
Four or more points can define a volume; one or more volumes can be defined. A marker can indicate a volume, the size of a volume, a selected volume, the cumulative size of selected volumes, and any combination thereof. Any combination of distance, angle, area and volume can be implemented. Any means known in the art of measuring distance, angle, area, volume known in the art can be implemented. Non-limiting examples of such measurement means include the methods typically found in Computer Aided Design (CAD) systems.
Distance, angle, area and volume measurements are typically 3D measurements. In some embodiments, the distance marker can give the distance between the end points as a triple of values, typically the three distances (x, y, z) of a Euclidean coordinate system. Other typical coordinate systems include, but are not limited to, cylindrical coordinate systems (r, Θ, z) and spherical coordinate systems (r, θ, φ).
At least two fixed points can indicate a preferred path. Two fixed points can mark the beginning and end of a path; additional fixed points can mark locations in the path. A path need not be a straight line. In some embodiments, a path can bypass obstacles. In some embodiments, a path can take into account non-flatness of, for example, an organ, if the path is to follow the surface of an organ. In some embodiments, a path can take into account movement such as, for non-limiting example, movement of an organ. In some embodiments, a path can take into account a desired distance between two tools at a point in a path, a desired orientation of at least one tool at at least one point in a path, and any combination thereof. Non-limiting examples include: if the path is that of a retractor, the end points of the path can define the distance between the blades of a retractor; a path can be defined so as to prevent collision between a tool and an organ; a path can be defined so as to prevent collision between two tools; a path can be defined so as to follow the contours of an organ; a path can be defined to follow contours of a lesion; and a path can be defined so as to avoid an item such as a nerve or blood vessel. A path can be stored in a database, can be displayed and any combination thereof.
In some embodiments, a path is constrained to go through fixed points marking the path; in some embodiments, a path can deviate from fixed points in order to avoid an obstacle.
In some embodiments, at least one path is calculated in 2D. In preferred embodiments, at least one path can be calculated in true 3D, enabling optimization of at least one of: the path, speed of movement of a surgical object during a procedure, position and orientation of a surgical object during a procedure, and avoiding collision with an item during a procedure. Any of the above can also be modified in real time. Such optimization and such modification can enable a surgical object to, for non-limiting example, avoid obstacles, move more slowly near items such as organs, prevent collision between tools, and any combination thereof. Path calculation can also include use of information on a stage reached in a procedure and/or use of stored information on preferences of the operator, based on stored information about the methods of a particular operator.
In preferred embodiments of the system, sufficient depth information is provided so that the position and orientation of at least one item in the field of view can be determined in true 3D, enabling the accurate determination of a distance between at least two items, an angle between at least two items, an area between at least two items, a volume between at least two items and any combination thereof.
The 3D position and orientation of an item can be determined from image analysis, by using data from multiple cameras, from position sensors attached to tools, from position sensors attached to tool manipulators, from "dead reckoning" of tool positions and orientations coming from position and orientation commands to tool manipulators, and any combination thereof.
From the accurate determination of distance and angle in 3D, an accurate determination can be made as to whether a tool's position, orientation, speed, acceleration, smoothness of motion and other parameters are correct. It is also possible to determine if a tool is accurately following a desired path, whether a collision can occur between two items, and whether the distance between two items is small enough that one or both can be activated.
An item that can be activated or deactivated based on distance information can include, but is not limited to, an ablator, a gripper, a fluid source, a light source, a pair of scissors, and any combination thereof.
For non-limiting example, activation of an ablator is best delayed until the ablator is close to the tissue to be ablated so that heating does not occur away from the tissue to be ablated, to minimize the possibility of damage to other tissue. With 3D position information, the ablator can be automatically activated when the distance between the ablator and the tissue to be ablated is less than a predetermined distance, so that there is no unnecessary heating of fluid or tissue away from the tissue to be ablated and so that ablation is carried out efficiently.
If only 2D distance information is available, an ablator could be activated when the 2D distance was small, but the distance perpendicular to the 2D plane (upward) was still large. In this case, the operator could be ignorant of this until it was observed that the ablator was heating fluid rather than ablating tissue. The operator would then have to move the ablator downward until ablation could occur, but would not have, nor could he be given, information on how far downward to move. At this point, either the ablator could be deactivated and moved until it contacted the tissue, or the ablator could be left activated until ablation began. In either case, unwanted damage to the tissue is likely.
In embodiments with a grid, preferably at least one distance marker is provided so that the operator knows the scale of the grid.
In some embodiments, an orientation indication can be provided, a horizon can be marked and any combination thereof. The orientation indication can be based on at least one item in a FOV such as, but not limited to, an organ, can be based on "dead reckoning", can be a direction relative to a region of interest, can be based on a position in a tool maneuvering system, can be based on a tool position determinable from a sensor signal, and any combination thereof.
Orientation can be determinable by providing a known orientation at a start of a procedure, by entering an orientation at a start of a procedure, by recognition of an orientation marker attached to a patient or to an operating table, and any combination thereof.
An orientation indication can allow an operator to remain aware of the orientation of a display view relative to a region of interest in the body, whatever the relative orientations of the body and the display view.
In preferred embodiments, an orientation marker remains within a fixed region in the display view.
A non-limiting example of an orientation marker is axes of a 3D coordinate system, with the axes labeled so that the identity of each axis is discernable at a glance. The axes are in a corner of the display view and rotate as the orientation of the display view changes.
Another non-limiting example of an orientation marker comprises an arrow with a fixed center, the direction of the arrow indicating a fixed (3D) direction in space. The point of the arrow will rotate around the center as the display view changes, while the color or texture of the arrow indicates whether the fixed direction is above or below the plane of the display image and the length of the arrow indicates the angle between the fixed direction and the plane of the display view.
Any means of indicating orientation known in the art can be used. In some embodiments, at least one point in an FOV can be marked. A markable point can be a critical point, indicate an organ or tissue, be a location on an organ or tissue, be a location within the body not on an organ or tissue, indicate a tool or other surgical object (such as a swab) introduced by an operator, or be a location (such as a tool tip) on a tool or other surgical object.
Sets of points, such as but not limited to a set of critical points or a set of points forming the outline of an item or the surface of an item can also be marked. A non-limiting example of an outline would be a line indicating the approximate extent of a tumor.
Marking can be by means of identifying a point in a display, which can be a 2D display or a 3D display; identifying a symbol representing an item, directing an indicator to a location by means of gestures or predetermined sounds, any other means known in the art of specifying a desired point, and any combination thereof. Identification can be by means of touching a point or item, touching an image of a point or item, pointing at a point or item, pointing at an image of a point or item and any combination thereof.
After marking, a point can be labeled; with the point indicated in an image by a virtual marker. A virtual marker can comprise any means of labeling images known in the art. Non- limiting examples of virtual markers include a predetermined geometrical shape, a predetermined word, a line encircling the image of a selected item, highlighting of the selected item (placing a patch of predetermined color or predetermined texture), and any combination thereof. Color-coding, with different colors indicating different types of virtual marker, can be used, either alone or in combination with any of the virtual markers described above.
In some embodiments, a virtual marker can indicate a selectable display view. In such embodiments, selection of a marker automatically alters the display view to the view specified by the marker. Such selectable display view markers can comprise, for non- limiting example, an outline of the selectable view, a point at the center of the selectable view, a patch or different color or texture covering the selectable view, and any combination thereof.
In some embodiments, portions of the image are enhanced, typically in order to be seen or identified more easily. An object which can be enhanced can include, but is not limited to, a blood vessel, a nerve, an organ, a ligament, a bone, a muscle, a lesion, a suspect location, a surgical tool, and any combination thereof. For the at least a portion of at least one image being enhanced, enhancement can include, but is not limited to, increasing brightness, altering color, applying at least one color patch, applying at least one texture patch, applying a label, recoloring, and any combination thereof.
A marker can comprise a distance measurement, an angle measurement, an area measurement, a volume measurement and any combination thereof. Two or more points can define a distance; multiple distances can be selected, both contiguous and non-contiguous. A marker can indicate a point, a distance, a path between points, a straight-line distance and any combination thereof. A numerical marker can indicate a total path length, a total straight-line distance, at least one pairwise distance, and any combination thereof.
A critical point can be used for monitoring such as, but not limited to, safety monitoring. Change in at least one critical point can indicate a change in performance, either of equipment or of an operator. A change in performance related to the equipment can be flagged up and a procedure stopped or changed, or at least one correction can be applied to at least one movement to maintain a procedure within limits of safety. Similarly, an operator's performance can be monitored and at least one warning can be flagged up if an operator's performance falls below a predetermined level of safety.
At least a portion of at least one procedure can be stored as a record; the record can comprise a complete procedure or can be an edited portion thereof, typically a surgical task or an identifiable unit. A record can comprise an image, an identifier, a fixed point, a critical point, a path and any combination thereof. Preferably, a record is tagged to enhance and simplify searching libraries of stored procedures, where a tagged identifier, image and any combination thereof is searchable.
A stored procedure can be a manually-executed procedure, an automatically-executed procedure, an autonomically-executed procedure and any combination thereof.
For non-limiting example, a complete procedure can comprise suturing an incision with a plurality of sutures. A suturing procedure typically comprises a plurality of repetitions of the following identifiable units: (1) moving a suturing needle (held in a first grasper) and a second grasper to a location for a suture, (2) passing the suturing needle through the tissue, (3) using both graspers, tying a knot, and (4) cutting the suturing thread. One record could comprise the complete procedure, the complete set of repetitions of steps (1) to (4); one record could comprise a surgical task, a single repetition of steps (1) to (4); one record could comprise an identifiable unit, a single one of the steps (1) to (4) and any combination thereof. A parameter can be a 2D position, for non-limiting example, in the plane of the field of view, of at least a portion of the at least one item; a 2D orientation, for non-limiting example, in the plane of the field of view, of at least a portion of the at least one item; a 3D position of at least a portion of the at least one item; a 3D orientation of at least a portion of the at least one item; a 2D projection of a 3D position of at least a portion of the at least one item, a velocity of at least a portion of the at least one item, an acceleration of at least a portion of the at least one item, an angle of at least a portion of the at least one item, a state of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
An identifier can include, but is not limited to, an identifier of an operator, type of procedure, a previous procedure during a surgical operation, a parameter, an identifier for an operating room, a physical characteristic of an operating room (e.g., temperature, humidity, time and date of cleaning, cleaning procedure, cleaning materials, type of lighting), a date of the procedure, a time and day of the week of a procedure, a duration of a procedure, a time from start of a previous procedure until start of a procedure, a time from end of a procedure until start of a subsequent procedure, an identifier of a patient, a physical characteristic of a patient, an outcome of a procedure, a type of malfunction during a procedure, severity of malfunction during a procedure, start time of malfunction, end time of malfunction, length of hospital stay for a patient, a readmission for a patient, and any combination thereof.
A non-limiting physical characteristic of a patient can include: age, height, weight, body mass index, health status, medical status, and any combination thereof. Tagging can be manual or automatic. For non-limiting example, typically, names of operators will be entered manually. In another non-limiting example, a critical point or a fixed point can be tagged manually or automatically. For non-limiting example, manual tagging can be by an operator indicating, by word, by gesture, or by touching a touchscreen, that a given point, such as the current position of a tool, is to be tagged as a critical point or a fixed point. For non-limiting example, automatic tagging can occur when a system identifies a point as a critical point or a fixed point.
An outcome of a procedure can have one or more aspects. Non-limiting examples of an aspect include: amount of bleeding after completion of a procedure, amount of bleeding during a procedure, return of an abnormality such as a tumor, speed of healing, adhesions and patient discomfort. For each of these exemplary aspects, a successful aspect would constitute: minimal bleeding after completion of a procedure, minimal bleeding during a procedure, no return of the abnormality, rapid healing, no adhesions and minimal patient discomfort. For each aspect, a partially successful aspect would constitute: some bleeding after completion of a procedure, some bleeding during a procedure, minimal return of the abnormality, moderately rapid healing, a few small adhesions and some patient discomfort. For each aspect, a partial failure in the aspect would constitute: significant bleeding after completion of a procedure, significant bleeding during a procedure, return of a significant amount of the abnormality, slow healing, significant adhesions and significant patient discomfort. For each aspect, complete failure in the aspect would constitute: serious or life-threatening bleeding after completion of a procedure, serious or life-threatening bleeding during a procedure, rapid return of the abnormality, very slow healing or failure to heal, serious adhesions and great patient discomfort. It is clear that an outcome can include any combination of aspects. For the exemplary aspects above, a procedure could have minimal bleeding, both during and after the procedure (successful) with a few adhesions (partial success), but significant patient discomfort (partial failure) and rapid return of the abnormality (complete failure).
At least one stored record of at least one procedure can become part of at least one "big data" analysis, A big data analysis can be, for non-limiting example, for a procedure, for an operator, for a hospital or medical center, for a surgical object, for a robotic maneuvering system and any combination thereof.
In some embodiments, the system can perform at least one procedure automatically, without an operator's intervention.
A non-limiting example of automatic control of a tool is control of zooming so that an operator has an optimum view during suturing. With automatic control, the laparoscope adjusts position and focus without operator intervention; the laparoscope zooms in during the tying process, zooms out after a suture has been completed to allow a better overview of the site, and follows the suturing tools as they are moved to the site of a next suture.
In some embodiments, the system can perform at least one procedure autonomically, without an operator's intervention.
A non-limiting example of autonomic functioning is an extension of the above, where the system carries out a suturing process without operator intervention, including moving the suturing tools, tying the sutures and cutting the suture threads, and moving an imaging device so that the process is observable.
In some embodiments, an override facility is provided, so that an operator can intervene manually. Manual intervention, via a manual override, can occur, for non-limiting example, if an event occurs that requires immediate action.
In some embodiments, the system can have different operation modes, depending on the identified procedure or the viewed scene.
In some embodiments, the system identifies surgical tools in the working area; in some embodiments, objects such as organs, lesions, bleeding and other items related to the patient are identified, smoke, flowing fluid, and the quality of the lighting (level, dark spots, obscured spots, etc.). If smoke, flowing fluid or bleeding is identified, the system can respond by e.g., virtual smoke or fogging removal (removal of smoke of fogging from an image via software), increasing a lighting level, providing light from an additional direction or angle, starting smoke or fog removal measures such as flowing fluid across a lens or through an area obscured by smoke or fog, starting suction, alerting an operator to the bleeding, clarifying an image either in software or my changing zoom or focus, applying adaptive optic correction, and any combination thereof.
In some embodiments, software -based super-resolution techniques can be used to sharpen at least one image without physically changing zoom or focus of an endoscope.
In some embodiments, at least one panoramic view is providable, either from a wide-angle lens or from unification of a plurality of images. In either case, software correction can be applied to provide a user with an undistorted image of a field of view, or an image with a desired degree or type of distortion.
A panoramic view can provide at least a 180° view and preferably it can provide a substantially 360° view. In some embodiments, at least one image can be stabilized via software, the stabilization removing shake or other unwanted movement, thereby increasing clarity.
In some embodiments, not all of the field of view of the lens is displayed. In such embodiments, either a wide-angle lens or a series of images of different regions of the operating field provide the system with a view of substantially all of an operating field. This enables the system to track objects outside of the region of the displayed image. A tracked object can be, but is not limited to: a surgical tool, an item of interest such as, but not limited to, bleeding, heart rate via pulsation of a blood vessel, unexpected movement of tissue, and any combination thereof. The system can perform zooming, can move the center of the field of view, or can display an alert or a popup to show or to follow a tracked object outside of the displayed image, or to show or to follow an item of interest. Preferably, the zooming or movement of the center of the FOV will be virtual zooming or movement, with no physical movement of the laparoscope. In some embodiments, control of movement of a surgical object can include a member of a group consisting of: changing arm movement and trajectory according to the FOV, changing velocity of movement according to the amount of zoom, closeness to an obstacle or stage in a procedure, and any combination thereof. Preferably, a rule-based approach will be used to determine movement or changes thereof.
Images from other modalities, such as, but not limited to, MRI images, CT images, ultrasound images, X-ray images, fluoroscopic images, molecular images, scintigraphy, SPECT, positron emission tomography (PET), other types of tomography, elastography, tactile imaging, photoacoustic imaging, thermography, Functional near-infrared spectroscopy (FNIR) and any combination thereof. Images from other modalities can be stored or realtime.
In some embodiments, at least one device controllable by the system is bed-mounted. In preferred embodiments, this reduces the footprint of the system over the patient.
In some embodiments, the system comprises control of a laparoscope. In some variants of these embodiments, the laparoscope has a wide-angle lens, preferably a high-definition lens. In some variants of these embodiments, the laparoscope is an articulated laparoscope; the system can comprise both a wide angle-lens and an articulated laparoscope. In some embodiments with a wide-angle lens, the displayed field of view can be controlled by movement of the laparoscope, by virtual FOV control (computer control of the FOV by altering the displayed portion of the image), and any combination thereof. In some embodiments, at least one tool can be automatically tracked by the system.
In preferred embodiments, there is full automation of the control of the robot arms positioning the laparoscope in at least two degrees of freedom, and preferably in all 7 degrees of freedom.
In some embodiments, the robotic arms are snake-like robotic arms providing full control by visual servoing (adaptive control via aimage analytics). This enables closed-loop control of all DOF's and, therefore, closed loop control of locating the target. Closed loop control also enables optimization by building an adaptive kinematic model for control of the robotic arms.
In addition to, or in place of, image analysis, at least a portion of at least one surgical object can be tracked using the kinematics of the robot. In such systems, a 3D pose of a surgical object can be calculated using known or measured joint angles of the robot, the parameters of the system and a keyhole-based (due to the trocar point of surgical insertion) forward kinematics algorithm. In preferred embodiments, the system can be in communication with at least one other device or system. For non-limiting example, the control software can control at least one surgical object, can be in communication with at least one advanced imaging system, can function as part of an integrated operating room and any combination thereof.
In some embodiments, the control software can have full connectivity with a member of a group consisting of: digital documentation, PACS, navigation, other health ΓΤ systems, and any combination thereof.
In addition to or in place of analyzing images of the surgical environment, at least one sensor can be used to provide information on location, orientation, speed, acceleration, and force exerted by or on at least one surgical tool.
A sensor (such as, for non-limiting example, at least one motion sensor) can be in mechanical communication with at least one tool; it (such as, for non-limiting example, a RFID tag) can be in wired communication with the tool; it can be in wireless communication with the tool; at least one processor can determine movement of at least one tool by determining change in position of at least one robot arm; and any combination thereof.
A sensor can be an accelerometer, a motion sensor, an EVIU, a sensor wearable by an operator, a sensor attachable to an item, an RFID tag attachable to an item, an ultrasound sensor, an infrared sensor, a CT image, an MRI image, an X-ray image, and any combination thereof. Preferably, one IMU can be mounted internally, in conjunction with a surgical tool such as a laparoscope, with a second IMU mounted on a movement-control robot, which improves the robustness of a determination of the position, orientation and movement of the surgical tool.
An item in a field of view of at least one image can be tracked by identifying at least one inherent, distinguishing characteristic of the item. For example, this can include the shape, color, texture, and movement of an item. To enhance tracking, an item can be modified to make it more recognizable in the camera image. For instance, a colored marker, a tracking pattern and any combination thereof can be affixed to at least one surgical object to aid in detection of the surgical object by a computer algorithm or to aid in providing an instruction to an operator.
Other tracking technologies include sensor-based technologies, including optical sensor technologies, electromagnetic sensor technologies and ultrasound sensor technologies. Optical techniques, such as an infrared tracking system, can locate an object that has at least one infrared marker attached to it. The object being tracked does not require any wires, but the line of sight from the tracking system to the tracked object must be kept clear.
In a sensor-based technologies, the sensor can be, for non-limiting example, an electromagnetic sensor; an ultrasound sensor; an inertial sensor to sense the angular velocity and the acceleration of the tool or other item; a gyroscope, an accelerometer, an IMU and any combination thereof.
An electromagnetic tracking system can also be used to locate a surgical object. At least one magnetic sensor can be affixed to a surgical object, and a magnetic transmitter emits a field that a magnetic sensor can detect.
A plurality of sensors can be combined into a single unit, such as an Inertial Measurement Unit (IMU). Typically, an IMU incorporates sensors such as an accelerometer, a gyroscope, a magnetometer and any combination thereof to track at least one of the orientation, position, and velocity of a surgical object. An IMU can measure and report specific force, angular rate, and, if a magnetometer is included, magnetic field. An IMU can be designed to transmit data wirelessly. An IMU can be used to provide dead-reckoning control of objects. However, IMUs can experience increasing error over time (especially in position), and some of the sensors can be sensitive to interference from other devices in an operating room. In some embodiments of autonomic robotic control of surgical tools, two IMUs are used for dead reckoning, one at the base, the proximal end, of a flexible robotic arm and one at or near the distal end of the flexible robotic arm. The distal IMU can be mounted to a laparoscope or other surgical tool or it can be mounted on the distal end of the flexible robotic arm itself. In some embodiments, other IMUs are provided at intermediate positions along the flexible robotic arm. Fig. 11A schematically indicates an embodiment of a flexible robotic arm (500) which includes proximal and distal IMUs (510). Fig. 11B schematically indicates an embodiment of a flexible robotic arm (500) with intermediate IMUs (515) in addition to the proximal and distal IMUs (510).
An optimum (second) parameter is typically derived from a previous procedure. The procedure can be: a manually-executed procedure, an automatically-executed procedure, an autonomically-executed procedure, a simulation of a procedure and any combination thereof. A manually-executed procedure was preferably executed by an experienced surgeon. An automatically-executed procedure is one in which one surgical tool automatically responds to a movement or other action of a second surgical tool. Non-limiting examples of automatically-executed procedures include tracking of a surgical tool by an endoscope or changing a lighting level in response to an increase in a perceived amount of smoke. An autonomically-executed procedure is one which can be executed independently of actions of a surgeon or of other tools. Non-limiting examples of autonomic procedures include executing a complete suture and executing a plurality of sutures to close an incision. A simulated procedure can be a procedure carried out by an operator on a simulator or a procedure automatically or autonomically executed on a simulator or using simulated or stored data.
A prestored parameter can also be generated by averaging, either by averaging a plurality of procedures and extracting a single parameter from the averaged procedure, or by generating a parameter for each procedure and then averaging the generated parameters. Kinematic tracking can be used to determine at least one property of at least one surgical object for a surgical object under robotic control. A typical robotic system includes one or more jointed arms that can manipulate at least one surgical object on behalf of an operator. The robot arm can also include at least one sensor (such as, but not limited to, an encoder, a potentiometer, a motion sensor, and an accelerometer) that can accurately determine the state of each joint. If the fixed properties of the physical structure of the robot arm are known (lengths of links, twists, etc.), they can be combined with the dynamic joint values to form a mathematical model of the robot arm. Desired surgical object properties, such as the position and orientation of the end-effector, the surgical tool, can be computed from this model, for non-limiting example, by forward kinematics.
Inverse kinematics can be used to determine, from the position and orientation of a surgical tool, the positions and orientations of the joint parameters that provided that position of the surgical tool.
There are many techniques for tracking at least one item in an image, where the camera can operate in the infrared (IR), in the visible, in the UV, and any combination thereof. An item can be tracked by identifying at least one inherent, distinguishing characteristic of the item. For non-limiting example, the distinguishing characteristic can be the shape, color, texture, movement and any combination thereof of an item. To enhance tracking, an item can be modified to make it more recognizable in an image. For instance, a colored marker, a tracking pattern, an LED and any combination thereof can be affixed to at least one surgical object to aid in detection by a computer algorithm or to aid in providing an instruction to an operator. A minimum of three non-collinear markers is necessary for determining six DOF, if the sole means of determining surgical tool location and orientation is the marker. Although a tracking system using at least one marker can provide high accuracy and reliability, it depends on a clear line of sight between the tracked tool and the camera system.
At least one LED can also be used to measure distance between a surgical object and tissue, typically by reflecting from tissue light emitted by an LED attached to a surgical object.
Any or all of the tracking means above can be wholly or partially inside the body, outside the body and any combination thereof. For non-limiting example, a tool can have a marker attached near its handle (outside the body) and a colored patch near its tip (inside the body). In this non-limiting example, movement of the marker is tracked by a camera outside the body (outside-outside), while movement of the colored patch is tracked by a camera inside the body (inside-inside). In another non-limiting example, an EM emitter can be close to the tip of the tool, while the EM sensor is attached to the operating table (inside-outside). Other combinations will be obvious to one skilled in the art.
An electromagnetic (EM) tracking system can also be used to locate at least one surgical object. By computing the position and orientation of at least one electromagnetic receiver on a surgical object as it moves through space, a dynamic, real-time measurement of the position and orientation of a surgical object can be found.
In some embodiments, the electromagnetic receivers are attached to an operator's hands, tracking the movements (changing position and orientation) of the tools by tracking the movements of the hands. However, keeping the sensors in a stable position during the entire execution of a surgical procedure can be difficult. Furthermore, the movements of the surgeon's hands need not be directly related to movements of a surgical tool.
Electromagnetic tracking systems do not need a clear line of sight, but are strongly affected by ferromagnetic objects, such as steel tools or electronic equipment in the clinical environment, which can seriously degrade tracking accuracy by affecting the local magnetic fields. Moreover, the need for wires in these systems can interfere with the use of laparoscopic instruments.
Combined methods can also be used, for non-limiting example, a combination of at least one passive optical marker and at least one EM sensor on a tool in order to minimize the effects of occasional blocking of the line-of-sight of the optical markers and distortion in the EM system. In addition, at least one force/torque sensor can be mounted on at least one surgical object. This exemplary combination can accurately measure position, orientation, velocity, acceleration, motion smoothness, and force applied by the tool.
Another combined method integrates on at least one surgical object at least one inertial sensor and at least one electromagnetic sensor. The inertial sensor, typically a gyroscope, accelerometer, velocity sensor and any combination thereof, measures the angular velocity and the acceleration of the target. These motion parameters are used to compute the position and orientation of the tracked object. The main advantage of this device is the high update rate, but it suffers from error accumulation over time, resulting in larger errors in velocity and position, which, in some clinical applications, can not be tolerated. Rectification algorithms can be applied in order to reduce the effects of error accumulation.
In preferred embodiments, from the locations and movements, the system can determine which procedure is being executed and the critical points of the procedure. In less-preferred embodiments, the procedure is input to the system by an operator.
Ultrasound can be used in much the same manner as optical tracking. Commonly, three or more emitters are mounted on a surgical object. Each emitter generates a sonic signal that is detected by a receiver placed at a fixed known position in the environment. Based on the sonic signals generated by the emitters, the system can determine their positions by triangulation. Combining three receivers, the tracker can estimate also the orientation of the target. However, these systems suffer from the environment-dependent velocity of the sound waves, which varies with temperature, pressure and humidity. The loss of energy of the ultrasonic signal with distance also limits the range of tracking. In addition, acoustic tracking requires line-of- sight, lack of which can affect the quality of the signal.
In some embodiments, a restricting mechanism is provided, which can restrict movement of an endoscope, at least one said surgical tool, and any combination thereof. The restricting mechanism can be, for non-limiting example, a brake, a stop, a catch, software configured to restrict movement, any other means of restricting movement known in the art, and any combination thereof.

Claims

CLAIMS:
1. A system for identifying at least one critical point in a procedure, comprising:
a. at least one imaging device configured to provide, for any given time t, at least one first image in a field of view of a surgical environment and, for at least one time t + At, to provide at least one second image in said field of view; b. at least one processor in communication with said at least one imaging device; said at least one processor is configured to (z) analyze said at least one first image; (z'z) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (Hi) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t, (z'v) identify from at least one second image said at least one item, and, (v) calculate from said least one second image, for said at least one item, said at least one first parameter at said time t + At; and,
c. at least one communicable database configured to (z) store said at least one first parameter at said time t; and (z'z) store said at least one first parameter at said time t + At;
wherein, from comparison between said at least one first parameter at said time t and said at least one first parameter at said time t + At, said at least one critical point is identifiable.
2. The system of claim 1, wherein said database comprises at least one prestored parameter for any given time t.
3. The system of claim 2, wherein said at least one processor is configured to (z) compare said at least one first parameter at said time t to said at least one prestored parameter at said time t; and (z'z) identify, from said comparison, at least one said critical point.
4. The system of claim 3, wherein, upon detection of a difference between said at least one first parameter and said at least one prestored parameter less than a predetermined value, a critical point is determinable.
5. The system of claim 4, wherein said predetermined value is in a range of about 0.1% to about 15%.
6. The system of claim 4, wherein said predetermined value is about 5%.
7. The system of claim 2, wherein said prestored parameter is calculable from a third image for said time t in a procedure selected from a group consisting of: said procedure executed at a different time, said procedure executed on a simulator; said procedure simulated in a processor, and any combination thereof.
8. The system of claim 1, wherein said surgical procedure is selected from a group consisting of an identifiable unit, a surgical task, a complete procedure, and any combination thereof.
9. The system of claim 1, wherein, upon detection of a difference between said at least one first parameter at said time t and said at least one first parameter at said time t + At is greater than a predetermined value, a critical point is determinable.
10. The system of claim 9, wherein said predetermined value is in a range of about 0.1% to about 15%.
11. The system of claim 9, wherein said predetermined value is about 5%
12. The system of claim 1, wherein said comparison is a real-time comparison.
13. The system of claim 1, wherein said at least one critical point is selectable from a group consisting of: a location in said surgical environment, a beginning of a procedure, an end of a procedure, an intermediate point in a procedure and any combination thereof.
14. The system of claim 13, wherein a plurality of said locations in said surgical environment are determinable from at least one said image.
15. The system of claim 14, wherein a path is determinable from said plurality of said locations.
16. The system of claim 16, wherein a member of an overlay group consisting of: said path, at least one said critical point, and any combination thereof is overlayable on at least one said image in said field of view.
17. The system of claim 17, wherein at least one member of said overlay group is storable in conjunction with said at least one said image.
18. The system of claim 1, wherein at least one indication is displayable upon determination of said at least one critical point.
19. The system of claim 18, wherein said at least one indication is selected from a group consisting of a restricting mechanism, a visible signal, an audible signal, a tactile signal and any combination thereof.
20. The system of claim 19, wherein said restricting mechanism is configured to restrict movement of a member of a group consisting of: at least one endoscope, at least one said surgical object and any combination thereof.
21. The system of claim 19, wherein said visible signal is selected from a group consisting of a constant-color pattern, a varying-color pattern, a constant- shape pattern, a varying- shape pattern, constant-size pattern, a varying-size pattern, an arrow, a word and any combination thereof.
22. The system of claim 19, wherein said audible signal is selected from a group consisting of a constant-pitch sound, a varying-pitch sound, a constant-loudness sound, a varying- loudness sound, a word and any combination thereof.
23. The system of claim 19, wherein said tactile signal is selected from a group consisting of a vibration, a constant-pressure signal, a varying-pressure signal, a stationary signal, a moving signal and any combination thereof.
24. The system of claim 23, wherein said tactile signal is applyable to a member of a group consisting of: a head, a neck, a torso, an arm, a wrist, a hand, a finger, a leg, an ankle, a toe and any combination thereof.
25. The system of claim 1, wherein said critical point for said at least one item is ascertainable from at least one second parameter derivable from a member of a group consisting of: a signal from a sensor, a forward kinematics calculation, an inverse kinematics calculation, a CT image, an MRI image, an X-ray image and any combination thereof.
26. The system of claim 25, wherein communication between said at least one sensor and said at least one item is selected from a group consisting of: mechanical communication, wired communication, wireless communication and any combination thereof.
27. The system of claim 25, wherein said sensor is selected from a group consisting of an electromagnetic sensor; an ultrasound sensor; an inertial sensor to sense the angular velocity and the acceleration of the tool or other item; an accelerometer, a motion sensor, an EVIU, a sensor wearable by an operator, a sensor attachable to an item, an RFID tag attachable to an item, an ultrasound sensor, an infrared sensor, gyro-meter, tachometer, shaft encoder, rotary encoder, strain gauge and any combination thereof.
28. The system of claim 25, wherein said at least one first parameter is selected from a group consisting of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, state of an item, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
29. The system of claim 25, wherein said at least one prestored parameter is selected from a group consisting of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, state of an item, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
30. The system of claim 25, wherein said at least one second parameter is selected from a group consisting of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, state of an item, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
31. The system of claim 25, wherein a member of a group consisting of said at least one first parameter, said at least one prestored parameter, said at least one second parameter, and any combination thereof is a function of time.
32. The system of claim 25, wherein said at least one critical point is identifiable in a manner selected from a group consisting of: a comparison of a plurality of consecutive first parameters to a plurality of consecutive prestored parameters, a comparison of a plurality of consecutive second parameters to a plurality of consecutive prestored parameters, and any combination thereof.
33. The system of claim 1, wherein said at least one imaging device is configured to real time provide a plurality of said images.
34. The system of claim 1, wherein said database is configured to store at least one record of at least one procedure, each said at least one record comprising a member of a group consisting of: said at least one image, said at least one first parameter, said at least one prestored parameter, said at least one second parameter, said at least one critical point, a fixed point, an object in said surgical environment, an angle, a distance, a reference grid, a horizon, an area, a volume, and any combination thereof.
35. The system of claim 34, wherein said at least one record is selectable based upon an identifier.
36. The system of claim 34, wherein said identifier is selected from a group consisting of: an identifier of an operator, an identifier of an operating room, a physical characteristic of an operating room, a start time of a procedure, end time of a procedure, duration of a procedure, date of a procedure, an identifier of a patient, a physical characteristic of a patient, an outcome of a procedure, length of hospital stay for a patient, a readmission for a patient, and any combination thereof.
37. The system of claim 36, wherein said physical characteristic of said operating room during said procedure is selected from a group consisting of: temperature, humidity, time of cleaning, date of cleaning, cleaning procedure, cleaning material, type of lighting and any combination thereof.
38. The system of claim 36, wherein said physical characteristic of said patient is selected from a group consisting of: age, height, weight, body mass index, health status, medical status, and any combination thereof.
39. The system of claim 34, wherein said outcome of said procedure is selected from a group consisting of: a successful aspect, a partially successful aspect, a partial failure in an aspect, a complete failure in an aspect, and any combination thereof.
40. The system of claim 1, wherein said at least one image is selected from a group consisting of: a 2D image, a 3D image, a panoramic image, a high resolution image, a 3D image reconstructed from at least one 2D image, a 3D stereo image and any combination thereof.
41. The system of claim 1, wherein overlayable on at least a portion of said at least one image is at least one of a group consisting of: said at least one critical point, a fixed point, a path, a suggestion, an instruction, a distance, an angle, an area, a volume, a size scale, information on a medical history of a patient, and any combination thereof is.
42. A system for identifying at least one critical point in a procedure, comprising:
a. at least one imaging device configured to provide, for any given time t, at least one first image in a field of view of a surgical environment;
b. at least one processor in communication with said at least one imaging device; said at least one processor is configured to (z) analyze said at least one first image; (z'z) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (Hi) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t, (iv) compare said at least one first parameter at said time t to at least one prestored parameter at said time t, and,
c. at least one communicable database configured to (i) store said at least one first parameter at said time t; and (ii) store said at least one prestored parameter for any given time t;
wherein, from comparison between said at least one first parameter at said time t and said at least one prestored parameter at said time t, said at least one critical point is identifiable.
43. The system of claim 42, wherein said at least one processor is configured to (i) provide, for at least one time t + At, at least one second image in said field of view; (ii) identify from at least one second image said at least one item; (Hi) calculate from said least one second image said at least one first parameter at said time t + At; (iv) store said at least one first parameter at said time t + At; and (v) identify, from said comparison, at least one said critical point.
44. A method for identifying at least one critical point in a procedure, comprising steps of: a. providing a system for identifying at least one critical point in a procedure, comprising:
i. at least one imaging device configured to provide, for any given time t, at least one first image in a field of view of a surgical environment and, for at least one time t + At, to provide at least one second image in said field of view;
ii. at least one processor in communication with said at least one imaging device; said at least one processor is configured to (i) analyze said at least one first image, (ii) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (Hi) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t; (iv) identify from at least one second image said at least one item, and (v) calculate from said least one second image, for said at least one item, said at least one first parameter at said time t + At; and,
iii. at least one communicable database configured to (z) store said at least one first parameter at said time t; and; and (n) store said at least one first parameter at said time t + At;
b. providing, via said at least one imaging device, said at least one first image at said time t and said at least one second image at said time t + At;
c. analyzing said at least one first image and determining, from said at least one first image, said at least one item;
d. calculating from said at least one first image, for said at least one item, said at least one first parameter at said time t;
e. storing said at least one first parameter at said time t;
f. analyzing said at least one second image and determining, from said at least one second image, said at least one item;
g. calculating from said at least one second image, for said at least one item, said at least one first parameter at said time t + At;
h. storing said at least one first parameter at said time t + At; and
i. comparing said at least one first parameter at said time t to said at least one first parameter at said time t + At,
thereby, from said comparison, identifying said at least one critical point.
45. The method of claim 44, additionally comprising a step of storing in said database at least one prestored parameter for any given time t.
46. The method of claim 45, additionally comprising steps of (z) comparing said at least one first parameter at said time t to said at least one prestored parameter at said time t; and (ii) identifying, from said comparison, at least one said critical point.
47. The method of claim 46, additionally comprising a step of determining a critical point by detecting a difference between said at least one first parameter and said at least one prestored parameter less than a predetermined value.
48. The method of claim 47, additionally comprising a steps of selecting said predetermined value to be in a range of about 0.1% to about 15%.
49. The method of claim 47, additionally comprising a step of selecting said predetermined value to be about 5%.
50. The method of claim 45, additionally comprising a step of calculating said prestored parameter from a second image for said time t in a procedure selected from a group consisting of: said procedure executed at a different time, said procedure executed on a simulator; said procedure simulated in a processor, and any combination thereof.
51. The method of claim 44, additionally comprising a step of selecting said surgical procedure is selected from a group consisting of an identifiable unit, a surgical task, a complete procedure, and any combination thereof.
52. The method of claim 44, additionally comprising a step of determining a critical point by detecting a difference between said at least one first parameter at time t and said at least one first parameter at time t + At greater than a predetermined value.
53. The method of claim 52, additionally comprising steps of selecting said predetermined value to be in a range of about 0.1% to about 15%.
54. The method of claim 52, additionally comprising steps of selecting said predetermined value to be about 5%.
55. The method of claim 44, additionally comprising a step of real-time performing said comparison.
56. The method of claim 44, additionally comprising a step of selecting said at least one critical point from a group consisting of: a location in said surgical environment, a beginning of a procedure, an end of a procedure, an intermediate point in a procedure and any combination thereof.
57. The method of claim 56, additionally comprising a step of determining a plurality of said locations in said surgical environment from at least one said image.
58. The method of claim 57, additionally comprising a step of determining a path from said plurality of said locations in said surgical environment.
59. The method of claim 44, additionally comprising a step of overlaying on at least one said image in said field of view a member of an overlay group consisting of: said path, at least one said critical point, and any combination thereof.
60. The method of claim 59, additionally comprising a step of storing at least one member of said overlay group in conjunction with said at least one said image.
61. The method of claim 44, additionally comprising a step of displaying at least one indication upon determination of said at least one critical point.
62. The method of claim 61, additionally comprising a step of selecting said indication from a group consisting of: a restricting mechanism, a visible signal, an audible signal, a tactile signal and any combination thereof.
63. The method of claim 61, additionally comprising a step of restricting, by means of said restricting mechanism, movement of a member of a group consisting of: at least one endoscope, at least one said item and any combination thereof.
64. The method of claim 61, additionally comprising a step of selecting said visible signal from a group consisting of a constant-color pattern, a varying-color pattern, a constant- shape pattern, a varying-shape pattern, constant-size pattern, a varying-size pattern, an arrow, a word and any combination thereof.
65. The method of claim 61, additionally comprising a step of selecting said audible signal from a group consisting of a constant-pitch sound, a varying-pitch sound, a constant- loudness sound, a varying-loudness sound, a word and any combination thereof.
66. The method of claim 61, additionally comprising a step of selecting said tactile signal from a group consisting of a vibration, a constant-pressure signal, a varying -pressure signal, a stationary signal, a moving signal and any combination thereof.
67. The method of claim 66, additionally comprising a step of applying said tactile signal to a member of a group consisting of: a head, a neck, a torso, an arm, a wrist, a hand, a finger, a leg, an ankle, a toe and any combination thereof.
68. The method of claim 44, additionally comprising a step of ascertaining said critical point for said at least one item from at least one second parameter derivable from a member of a group consisting of: a signal from a sensor, a forward kinematics calculation, an inverse kinematics calculation, a CT image, an MRI image, an X-ray image, and any combination thereof.
69. The method of claim 68, additionally comprising a step of providing communication between said sensor and said at least one item selected from a group consisting of: in mechanical communication, wired communication, wireless communication and any combination thereof.
70. The method of claim 68, additionally comprising a step of selecting said sensor from a group consisting of an electromagnetic sensor; an ultrasound sensor; an inertial sensor to sense the angular velocity and the acceleration of the tool or other item; an accelerometer, a motion sensor, an IMU, a sensor wearable by an operator, a sensor attachable to an item, an RFID tag attachable to a surgical object, an ultrasound sensor, an infrared sensor, gyro-meter, tachometer, shaft encoder, rotary encoder, strain gauge and any combination thereof.
71. The method of claim 68, additionally comprising a step of selecting said at least one first parameter from a group consisting of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, state of an item, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
72. The method of claim 68, additionally comprising a step of selecting said at least one prestored parameter from a group consisting of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, state of an item, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
73. The method of claim 68, additionally comprising a step of selecting a said at least one second parameter from a group consisting of: 2D position of at least a portion of at least one item, 2D orientation of at least a portion of at least one item, 3D position of at least a portion of at least one item, 3D orientation of at least a portion of at least one item, 2D projection of a 3D position of at least a portion of said at least one item, movement of at least a portion of at least one said item, energy use; idle time, approach time, speed, maximum speed, speed profile, acceleration, motion smoothness, path length, distance efficiency, depth perception, transit profile, deviation on horizontal plane, deviation on vertical plane, response orientation, economy of area (EOA), economy of volume (EOV), number of movements, force range, interquartile force range, integral of the force, integral of the grasping force, integral of the Cartesian force, first derivative of the force, second derivative of the force, third derivative of the force, lighting level, amount of suction, amount of fluid flow, state of an item, heating level in an ablator, amount of defogging, amount of smoke removal, activation of an item, deactivation of an item, bleeding, change in heart rate, change in blood pressure, change in color of an organ, and any combination thereof.
74. The method of claim 68, additionally comprising a step of providing a member of a group consisting of said at least one first parameter, said at least one prestored parameter, said at least one second parameter, and any combination thereof as function of time.
75. The method of claim 68, additionally comprising a step of identifying said at least one critical point in a manner selected from a group consisting of: comparing a plurality of consecutive first parameters to a plurality of consecutive prestored parameters, comparing a plurality of consecutive first parameters to a plurality of consecutive second parameters, comparing a plurality of consecutive second parameters to a plurality of consecutive prestored parameters and any combination thereof.
76. The method of claim 44, additionally comprising a step of providing, by means of said at least one imaging device, a plurality of said images.
77. The method of claim 44, additionally comprising a step of storing in said database at least one record of at least one procedure, each said at least one record comprising a member of a group consisting of: said at least one image, said at least one first parameter, said at least one prestored parameter, said at least one second parameter, said at least one critical point, a fixed point, an object in said surgical environment, an angle, a distance, a reference grid, a horizon, an area, a volume, and any combination thereof.
78. The method of claim 77, additionally comprising a step of selecting said at least one record based upon said identifier.
79. The method of claim 77, additionally comprising a step of selecting said identifier from a group consisting of: an identifier of an operator, an identifier of an operating room, a physical characteristics of an operating room, a start time of a procedure, end time of a procedure, duration of a procedure, date of a procedure, an identifier of a patient, a physical characteristic of a patient, an outcome of a procedure, length of hospital stay for a patient, a readmission for a patient, and any combination thereof.
80. The method of claim 79, additionally comprising a step of selecting said physical characteristic of said operating room from a group consisting of: temperature, humidity, time of cleaning, date of cleaning, cleaning procedure, cleaning material, type of lighting and any combination thereof.
81. The method of claim 79, additionally comprising a step of selecting said physical characteristic of said patient from a group consisting of: age, height, weight, body mass index, health status, medical status, and any combination thereof.
82. The method of claim 79, additionally comprising a step of selecting said outcome of said procedure from a group consisting of: a successful aspect, a partially successful aspect, a partial failure in an aspect, a complete failure in an aspect, and any combination thereof.
83. The method of claim 44, additionally comprising a step of selecting said at least one image from a group consisting of: a 2D image, a 3D image, a panoramic image, a high resolution image, a 3D image reconstructed from at least one 2D image, a 3D stereo image and any combination thereof.
84. The method of claim 44, additionally comprising a step of overlaying on said at least one image at least one of a group consisting of: said critical point, a fixed point, a path, a suggestion, an instruction, a distance, an angle, an area, a volume, a size scale, information on a medical history of a patient, and any combination thereof.
85. A method for identifying at least one critical point in a procedure, comprising steps of: a. providing a system for identifying at least one critical point in a procedure, comprising:
i. at least one imaging device configured to provide, for any given time t, at least one first image in a field of view of a surgical environment;
ii. at least one processor in communication with said i at least one imaging device; said at least one processor is configured to (i) analyze said at least one first image; (ii) identify from said at least one first image at least one item selected from a group consisting of: an object in said field of view, a surgical tool, a fixed point and any combination thereof, (iii) calculate from said at least one first image, for said at least one item, at least one first parameter at said time t, (iv) compare said at least one first parameter at said time t to at least one prestored parameter at said time t, and, iii. at least one communicable database configured to (i) store said at least one first parameter at said time t; and (ii) store said at least one prestored parameter for any given time t;
b. providing, via said at least one imaging device, said at least one image of a field of view at said time t;
c. analyzing said at least one image and determining, from said at least one image, said at least one item;
d. calculating from said at least one image, for said at least one item, said at least one first parameter at said time t;
e. storing said at least one first parameter at said time t;
f. analyzing said at least one second image and determining, from said at least one second image, said at least one item; and
g. comparing said at least one first parameter at said time t to said at least one prestored parameter at said time t,
thereby, from said comparison, identifying said at least one critical point.
86. The method of claim 85, additionally comprising steps of: (i) analyzing said at least one second image and determining, from said at least one second image, said at least one item; (ii) calculating from said at least one second image said at least one first parameter at said time t + At of said at least one item; (iii) storing said at least one first parameter at said time t + At; (iv) comparing said at least one first parameter at said time t to said at least one first parameter at said time t + At; and, (v) identifying, from said comparison, said at least one critical point.
EP16872548.9A 2015-12-07 2016-12-06 AUTONOMOUS SYSTEM FOR DETERMINING CRITICAL POINTS DURING LAPAROSCOPIC SURGERY Withdrawn EP3414737A4 (en)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US201562263749P 2015-12-07 2015-12-07
US201662290963P 2016-02-04 2016-02-04
US201662334460P 2016-05-11 2016-05-11
US201662336672P 2016-05-15 2016-05-15
PCT/IL2016/051306 WO2017098505A1 (en) 2015-12-07 2016-12-06 Autonomic system for determining critical points during laparoscopic surgery

Publications (2)

Publication Number Publication Date
EP3414737A1 true EP3414737A1 (en) 2018-12-19
EP3414737A4 EP3414737A4 (en) 2019-11-20

Family

ID=59013812

Family Applications (1)

Application Number Title Priority Date Filing Date
EP16872548.9A Withdrawn EP3414737A4 (en) 2015-12-07 2016-12-06 AUTONOMOUS SYSTEM FOR DETERMINING CRITICAL POINTS DURING LAPAROSCOPIC SURGERY

Country Status (2)

Country Link
EP (1) EP3414737A4 (en)
WO (1) WO2017098505A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023199357A1 (en) * 2022-04-13 2023-10-19 Garg Dr Suruchi A system of identifying plurality of parameters of a subject's skin and a method thereof

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3668437A2 (en) * 2017-08-16 2020-06-24 Covidien LP Method of spatially locating points of interest during a surgical procedure
US10758309B1 (en) * 2019-07-15 2020-09-01 Digital Surgery Limited Methods and systems for using computer-vision to enhance surgical tool control during surgeries
US11625834B2 (en) 2019-11-08 2023-04-11 Sony Group Corporation Surgical scene assessment based on computer vision
WO2021126786A1 (en) * 2019-12-16 2021-06-24 Intuitive Surgical Operations, Inc. Systems and methods for identifying and facilitating an intended interaction with a target object in a surgical space
US11896442B2 (en) * 2019-12-30 2024-02-13 Cilag Gmbh International Surgical systems for proposing and corroborating organ portion removals
WO2021202341A1 (en) * 2020-03-31 2021-10-07 Intuitive Surgical Operations, Inc. Systems and methods for facilitating automated operation of a device in a surgical space
WO2022005954A2 (en) * 2020-06-30 2022-01-06 Intuitive Surgical Operations, Inc. Systems and methods for tag-based instrument control
WO2022066810A1 (en) 2020-09-23 2022-03-31 Wayne State University Systems and methods for predicting and preventing bleeding and other adverse events
WO2022146911A1 (en) * 2021-01-04 2022-07-07 Intuitive Surgical Operations, Inc. Image-based seeding for registration and associated systems and methods
CN113133828B (en) * 2021-04-01 2023-12-01 上海复拓知达医疗科技有限公司 Interactive registration system, method, electronic device and readable storage medium for surgical navigation
CN113133829B (en) * 2021-04-01 2022-11-01 上海复拓知达医疗科技有限公司 Surgical navigation system, method, electronic device and readable storage medium
US20230346199A1 (en) * 2022-04-29 2023-11-02 3Dintegrated Aps Anatomy measurement
CN119033312B (en) * 2024-11-01 2024-12-27 北京云力境安科技有限公司 Control method, device, equipment and medium of soft endoscope control system

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR2920086A1 (en) * 2007-08-24 2009-02-27 Univ Grenoble 1 ANALYSIS SYSTEM AND METHOD FOR ENDOSCOPY SURGICAL OPERATION
EP2577641A4 (en) * 2010-05-26 2015-11-18 Health Research Inc METHOD AND SYSTEM FOR AUTOMATICALLY DETERMINING THE POSITION OF A TOOL FOR DRIVING TO LITTLE INVASIVE SURGICAL ACTS
WO2012060901A1 (en) * 2010-11-04 2012-05-10 The Johns Hopkins University System and method for the evaluation of or improvement of minimally invasive surgery skills
WO2013042107A1 (en) * 2011-09-20 2013-03-28 M.S.T. Medical Surgery Technologies Ltd. A device and method for maneuvering endoscope
US10553130B2 (en) * 2012-05-03 2020-02-04 Regents Of The University Of Minnesota Systems and methods for analyzing surgical techniques
US10744646B2 (en) * 2013-08-29 2020-08-18 Wayne State University Camera control system and method

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023199357A1 (en) * 2022-04-13 2023-10-19 Garg Dr Suruchi A system of identifying plurality of parameters of a subject's skin and a method thereof

Also Published As

Publication number Publication date
EP3414737A4 (en) 2019-11-20
WO2017098505A1 (en) 2017-06-15

Similar Documents

Publication Publication Date Title
EP3414737A1 (en) Autonomic system for determining critical points during laparoscopic surgery
JP7492506B2 (en) Navigation aids
US20200397515A1 (en) Interface for Laparoscopic Surgeries - Movement Gestures
US11638615B2 (en) Intelligent surgical tool control system for laparoscopic surgeries
US20210315645A1 (en) Feature identification
US12279841B2 (en) Robotic surgical safety via video processing
US20190008598A1 (en) Fully autonomic artificial intelligence robotic system
JP6293777B2 (en) Collision avoidance during controlled movement of image capture device and operable device movable arm
EP3413774A1 (en) Database management for laparoscopic surgery
JP2019524284A (en) Perform robot system movements
WO2017098506A9 (en) Autonomic goals-based training and assessment system for laparoscopic surgery
JP7395125B2 (en) Determining the tip and orientation of surgical tools
US12383341B2 (en) Systems and methods for automatically generating an anatomical boundary
WO2017098504A9 (en) Autonomic detection of malfunctioning in surgical tools
CN114828727A (en) Computer-assisted surgery system, surgery control device, and surgery control method
US12011236B2 (en) Systems and methods for rendering alerts in a display of a teleoperational system
Cheng et al. An augmented reality framework for optimization of computer assisted navigation in endovascular surgery
Bihlmaier Endoscope robots and automated camera guidance
US20230302646A1 (en) Systems and methods for controlling and enhancing movement of a surgical robotic unit during surgery
CN120351848A (en) Computer-aided distance measurement in surgical space
CN121889826A (en) Computer-aided registration and tracking of models of anatomical objects
GB2608016A (en) Feature identification
GB2611972A (en) Feature identification

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20181015

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G06T0007000000

Ipc: G16H0030400000

A4 Supplementary search report drawn up and despatched

Effective date: 20191018

RIC1 Information provided on ipc code assigned before grant

Ipc: G16H 20/40 20180101ALI20191014BHEP

Ipc: A61B 5/06 20060101ALI20191014BHEP

Ipc: A61B 17/00 20060101ALI20191014BHEP

Ipc: G16H 30/40 20180101AFI20191014BHEP

Ipc: A61B 90/30 20160101ALN20191014BHEP

Ipc: A61B 5/00 20060101ALI20191014BHEP

Ipc: A61B 34/20 20160101ALI20191014BHEP

Ipc: G16H 40/60 20180101ALI20191014BHEP

Ipc: G06T 7/246 20170101ALI20191014BHEP

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: TRANSENTERIX EUROPE SARL

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20230322

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20231003