diff options
| author | Dave Airlie <airlied@redhat.com> | 2026-08-01 07:06:31 +1000 |
|---|---|---|
| committer | Dave Airlie <airlied@redhat.com> | 2026-08-01 07:06:36 +1000 |
| commit | 82cb10c8bc506a4d2093e0e59672fd307f549d38 (patch) | |
| tree | 171e8a37136df6fb7f9ba25711592bfab68a32ec | |
| parent | 2b3f7c2c8dd8d710ab36f92a316f29fd37d83449 (diff) | |
| parent | 60693b580ab5f7f3158adcc2bab5ae816edd49ff (diff) | |
| download | linux-next-82cb10c8bc506a4d2093e0e59672fd307f549d38.tar.gz linux-next-82cb10c8bc506a4d2093e0e59672fd307f549d38.zip | |
Merge tag 'drm-xe-next-2026-07-30' of https://gitlab.freedesktop.org/drm/xe/kernel into drm-next
- Wait on external BO kernel fences in exec IOCTL (Brost)
- General clean-up (Anas)
- Documentation fix (Rafael)
- Add a debugfs for pcode information (Karthik)
- Free madvise VMA array on L2 flush failure (Guangshuo)
- Page Table related fixes (Shuicheng, Zongyao)
- Improvements GuC error handling and GuC small fixes (Sk, Zhanjun, Arvind)
- Add new W/As (Daniele, Harish)
- GuC paging engine support (Auld)
- Add and use more KLV helpers (Michal)
- Balance exec queue suspend/resume (Niranjana, Thomas)
- Fix BO prefetch with CONSULT_MEM_ADVISE_PREF_LOC (Himal)
- SRIOV: Disable display in admin only PF mode (Satya)
- Fix writable override for CRI NVM (Sasha)
- Fix VF CCS attach/detach race with in-flight BO moves (Brost)
- Introduce Xe Uncorrectable Error Handling (Riana)
- Fix WOPCM size for LNL+ (Daniele)
- Consolidate debugfs fault injection functions (Mallesh)
- Multi-queue related fixes and improvements (Niranjana, Jagmeet, Shuicheng)
- Add RAS GPU health indicator (Soham)
- PAT related improvements (Roper, Sanjay)
- NULL deref fix on migration on VF (Satya)
- i2c related fix (Raag)
- Fix SVM leak and clean up xe_vm_create (Shuicheng)
- Drop force_probe requirement for NVL-s (Gustavo)
- Optimise TT population for DONTNEED BOs (Auld)
- Add page size allocation mode control and coverage (Himal, Nareshkumar)
Signed-off-by: Dave Airlie <airlied@redhat.com>
From: Rodrigo Vivi <rodrigo.vivi@intel.com>
Link: https://patch.msgid.link/amt2kDVdyBK6VEyU@intel.com
88 files changed, 3714 insertions, 381 deletions
diff --git a/Documentation/ABI/testing/sysfs-driver-intel-xe-ras b/Documentation/ABI/testing/sysfs-driver-intel-xe-ras new file mode 100644 index 000000000000..3870e5a03a92 --- /dev/null +++ b/Documentation/ABI/testing/sysfs-driver-intel-xe-ras @@ -0,0 +1,30 @@ +What: /sys/bus/pci/drivers/xe/.../gpu_health +Date: July 2026 +KernelVersion: 7.3 +Contact: intel-xe@lists.freedesktop.org +Description: + This file exposes the current gpu health state and allows the gpu + health state to be updated. + + This sysfs file is present only on Intel Xe platforms that support + the gpu health indicator interface for RAS. Reading the current + health state is available to all users, while updating the health + state is restricted to administrative users only. + + Read returns a single line containing one of the valid values for + the current gpu health state. Writing one of the valid values + updates the current gpu health state. + + The valid values for the gpu health state are: + + ok + The gpu is healthy and operating within normal + parameters. + + warning + The gpu is experiencing minor issues but remains + operational. + + critical + The gpu is in a critical state and may not be + operational. diff --git a/Documentation/gpu/xe/xe_device.rst b/Documentation/gpu/xe/xe_device.rst index 39a937b97cd3..d3a022362ade 100644 --- a/Documentation/gpu/xe/xe_device.rst +++ b/Documentation/gpu/xe/xe_device.rst @@ -8,3 +8,10 @@ Xe Device Wedging .. kernel-doc:: drivers/gpu/drm/xe/xe_device.c :doc: Xe Device Wedging + +==================== +GPU Health Indicator +==================== + +.. kernel-doc:: drivers/gpu/drm/xe/xe_ras.c + :doc: GPU Health Indicator diff --git a/drivers/gpu/drm/xe/Kconfig.debug b/drivers/gpu/drm/xe/Kconfig.debug index 01227c77f6d7..79118d9efd93 100644 --- a/drivers/gpu/drm/xe/Kconfig.debug +++ b/drivers/gpu/drm/xe/Kconfig.debug @@ -86,6 +86,22 @@ config DRM_XE_KUNIT_TEST If in doubt, say "N". +config DRM_XE_DEBUG_PAGE_SIZE + bool "Enable debug control for user BO page-size allocation" + depends on DRM_XE_DEBUG && DEBUG_FS + help + Expose a debugfs knob to override user BO page-size allocation + handling for validation and debug. Supported modes include forced + 2M, forced 1G, and a mixed mode that exercises 4K, 64K, 2M, and + 1G page-size paths on platforms that support them. + + This is an unstable debugfs interface intended for development and + validation only. Its layout, contents, and existence may change or + be removed at any time with no regression warranty. + + Recommended for driver developers only. + If in doubt, say "N". + config DRM_XE_DEBUG_GUC bool "Enable extra GuC related debug options" depends on DRM_XE_DEBUG diff --git a/drivers/gpu/drm/xe/abi/guc_actions_abi.h b/drivers/gpu/drm/xe/abi/guc_actions_abi.h index 83a6e7794982..f5c9b37038d4 100644 --- a/drivers/gpu/drm/xe/abi/guc_actions_abi.h +++ b/drivers/gpu/drm/xe/abi/guc_actions_abi.h @@ -152,6 +152,7 @@ enum xe_guc_action { XE_GUC_ACTION_REPORT_PAGE_FAULT_REQ_DESC = 0x6002, XE_GUC_ACTION_PAGE_FAULT_RES_DESC = 0x6003, XE_GUC_ACTION_ACCESS_COUNTER_NOTIFY = 0x6004, + XE_GUC_ACTION_NOTIFY_UNCORRECTABLE_LOCAL_ERROR = 0x6005, XE_GUC_ACTION_TLB_INVALIDATION = 0x7000, XE_GUC_ACTION_TLB_INVALIDATION_DONE = 0x7001, XE_GUC_ACTION_TLB_INVALIDATION_ALL = 0x7002, diff --git a/drivers/gpu/drm/xe/abi/guc_capture_abi.h b/drivers/gpu/drm/xe/abi/guc_capture_abi.h index dd4117553739..ff9c0ae34a28 100644 --- a/drivers/gpu/drm/xe/abi/guc_capture_abi.h +++ b/drivers/gpu/drm/xe/abi/guc_capture_abi.h @@ -32,9 +32,10 @@ enum guc_capture_list_class_type { GUC_CAPTURE_LIST_CLASS_VIDEOENHANCE = 2, GUC_CAPTURE_LIST_CLASS_BLITTER = 3, GUC_CAPTURE_LIST_CLASS_GSC_OTHER = 4, + GUC_CAPTURE_LIST_CLASS_PAGING = 5, }; -#define GUC_CAPTURE_LIST_CLASS_MAX (GUC_CAPTURE_LIST_CLASS_GSC_OTHER + 1) +#define GUC_CAPTURE_LIST_CLASS_MAX (GUC_CAPTURE_LIST_CLASS_PAGING + 1) /** * struct guc_mmio_reg - GuC MMIO reg state struct diff --git a/drivers/gpu/drm/xe/abi/guc_klvs_abi.h b/drivers/gpu/drm/xe/abi/guc_klvs_abi.h index 644f5a4226d7..685c4ef17b73 100644 --- a/drivers/gpu/drm/xe/abi/guc_klvs_abi.h +++ b/drivers/gpu/drm/xe/abi/guc_klvs_abi.h @@ -22,6 +22,7 @@ * | | | - `GuC Scheduling Policies KLVs`_ | * | | | - `GuC VGT Policy KLVs`_ | * | | | - `GuC VF Configuration KLVs`_ | + * | | | - `GuC Reserved KLVs`_ | * | | | | * | +-------+--------------------------------------------------------------+ * | | 15:0 | **LEN** - length of VALUE (in 32bit dwords) | @@ -52,6 +53,12 @@ * _`GUC_KLV_GLOBAL_CFG_GROUP_SCHEDULING_AVAILABLE` : 0x3001 * Tells the driver whether scheduler groups are enabled or not. * Requires GuC ABI 1.26+ + * + * _`GUC_KLV_GLOBAL_CFG_NUM_PAGING_ENGINE_INSTANCES` : 0x3003 + * Tells the driver the paging engine configuration. + * Paging engine logical instances are guaranteed to be dense starting at + * index 0. + * Requires GuC ABI 1.36+ */ #define GUC_KLV_GLOBAL_CFG_GMD_ID_KEY 0x3000u @@ -60,6 +67,9 @@ #define GUC_KLV_GLOBAL_CFG_GROUP_SCHEDULING_AVAILABLE_KEY 0x3001u #define GUC_KLV_GLOBAL_CFG_GROUP_SCHEDULING_AVAILABLE_LEN 1u +#define GUC_KLV_GLOBAL_CFG_NUM_PAGING_ENGINE_INSTANCES_KEY 0x3003u +#define GUC_KLV_GLOBAL_CFG_NUM_PAGING_ENGINE_INSTANCES_LEN 1u + /** * DOC: GuC Self Config KLVs * @@ -154,6 +164,11 @@ enum { * (instead of waiting the full timeslice duration). The bit is instead set * to one if a single context is queued on the engine, to avoid it being * switched out if there isn't another context that can run in its place. + * + * _`GUC_KLV_OPT_IN_FEATURE_UNCORRECTABLE_LOCAL_ERROR_NOTIFICATION` : 0x4004 + * This flag will enable notification from GuC to KMD via G2H message + * GUC_ACTION_GUC2HOST_NOTIFY_UNCORRECTABLE_LOCAL_ERROR upon receiving the + * same interrupt from the CS. */ #define GUC_KLV_OPT_IN_FEATURE_EXT_CAT_ERR_TYPE_KEY 0x4001 @@ -162,6 +177,9 @@ enum { #define GUC_KLV_OPT_IN_FEATURE_DYNAMIC_INHIBIT_CONTEXT_SWITCH_KEY 0x4003 #define GUC_KLV_OPT_IN_FEATURE_DYNAMIC_INHIBIT_CONTEXT_SWITCH_LEN 0u +#define GUC_KLV_OPT_IN_FEATURE_UNCORRECTABLE_LOCAL_ERROR_NOTIFICATION_KEY 0x4004 +#define GUC_KLV_OPT_IN_FEATURE_UNCORRECTABLE_LOCAL_ERROR_NOTIFICATION_LEN 0u + /** * DOC: GuC Scheduling Policies KLVs * @@ -491,9 +509,10 @@ enum { #define GUC_KLV_VF_CFG_ENGINE_GROUP_PREEMPT_TIMEOUT_MIN_LEN 1u #define GUC_KLV_VF_CFG_ENGINE_GROUP_PREEMPT_TIMEOUT_MAX_LEN GUC_MAX_SCHED_GROUPS /* - * Workaround keys: + * Feature and Workaround keys: */ enum xe_guc_klv_ids { + GUC_FEATURE_KLV_DISABLE_MULTI_QUEUE = 0x5001, GUC_WORKAROUND_KLV_BLOCK_INTERRUPTS_WHEN_MGSR_BLOCKED = 0x9002, GUC_WORKAROUND_KLV_DISABLE_PSMI_INTERRUPTS_AT_C6_ENTRY_RESTORE_AT_EXIT = 0x9004, GUC_WORKAROUND_KLV_ID_GAM_PFQ_SHADOW_TAIL_POLLING = 0x9005, @@ -504,6 +523,24 @@ enum xe_guc_klv_ids { GUC_WA_KLV_RESET_BB_STACK_PTR_ON_VF_SWITCH = 0x900b, GUC_WA_KLV_RESTORE_UNSAVED_MEDIA_CONTROL_REG = 0x900c, GUC_WA_KLV_CLR_CS_INDIRECT_RING_STATE_IF_IDLE_AT_CTX_REG = 0x900e, + GUC_WA_KLV_REMAP_RANGED_TLB_INV = 0x900f, + GUC_WA_KLV_IGNORE_MMIO_READ_SEM_TOKEN_64 = 0x9010, }; +/** + * DOC: GuC Reserved KLVs + * + * Range of `GuC KLV`_ keys reserved for internal use by the GuC that will + * never be part of the offcial GuC ABI and can be reused by the drivers. + * + * Currently this range includes 1024 keys starting from: + * + * _`GUC_KLV_RESERVED_RANGE_START` : 0xF000 + * + * See `Xe Driver KLVs`_ for the KLVs that the Xe driver is currently using. + */ + +#define GUC_KLV_RESERVED_RANGE_START 0xf000u +#define GUC_KLV_RESERVED_RANGE_LEN 1024u + #endif diff --git a/drivers/gpu/drm/xe/abi/guc_scheduler_abi.h b/drivers/gpu/drm/xe/abi/guc_scheduler_abi.h index 19ec89bf39c5..85bba34277ed 100644 --- a/drivers/gpu/drm/xe/abi/guc_scheduler_abi.h +++ b/drivers/gpu/drm/xe/abi/guc_scheduler_abi.h @@ -21,7 +21,8 @@ #define GUC_BLITTER_CLASS 3 #define GUC_COMPUTE_CLASS 4 #define GUC_GSC_OTHER_CLASS 5 -#define GUC_LAST_ENGINE_CLASS GUC_GSC_OTHER_CLASS +#define GUC_PAGING_CLASS 6 +#define GUC_LAST_ENGINE_CLASS GUC_PAGING_CLASS #define GUC_MAX_ENGINE_CLASSES 16 #define GUC_MAX_INSTANCES_PER_CLASS 32 diff --git a/drivers/gpu/drm/xe/abi/xe_driver_klvs_abi.h b/drivers/gpu/drm/xe/abi/xe_driver_klvs_abi.h new file mode 100644 index 000000000000..3b557e56892a --- /dev/null +++ b/drivers/gpu/drm/xe/abi/xe_driver_klvs_abi.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2026 Intel Corporation + */ + +#ifndef _ABI_XE_DRIVER_KLVS_ABI_H +#define _ABI_XE_DRIVER_KLVS_ABI_H + +#include "abi/guc_klvs_abi.h" + +/** + * DOC: Xe Driver KLVs + * + * The Xe driver uses the following keys from the `GuC Reserved KLVs`_ range: + * + * _`MIGRATION_KLV_DEVICE_DEVID_KEY` : + * PCI device ID of the migrated VF. + * _`MIGRATION_KLV_DEVICE_REVID_KEY` : + * PCI device revision ID of the migrated VF. + */ + +#define MIGRATION_KLV_DEVICE_DEVID_KEY 0xf001u +#define MIGRATION_KLV_DEVICE_DEVID_LEN 1u +#define MIGRATION_KLV_DEVICE_REVID_KEY 0xf002u +#define MIGRATION_KLV_DEVICE_REVID_LEN 1u + +#endif diff --git a/drivers/gpu/drm/xe/regs/xe_pmt.h b/drivers/gpu/drm/xe/regs/xe_pmt.h index 240d57993ea6..a62ab05c6b4c 100644 --- a/drivers/gpu/drm/xe/regs/xe_pmt.h +++ b/drivers/gpu/drm/xe/regs/xe_pmt.h @@ -15,6 +15,8 @@ #define ENERGY_PKG REG_GENMASK64(31, 0) #define ENERGY_CARD REG_GENMASK64(63, 32) +#define PUNIT_VERSION_OFFSET 0xA0 + #define BMG_TELEMETRY_BASE_OFFSET 0xE0000 #define BMG_TELEMETRY_OFFSET (SOC_BASE + BMG_TELEMETRY_BASE_OFFSET) diff --git a/drivers/gpu/drm/xe/regs/xe_regs.h b/drivers/gpu/drm/xe/regs/xe_regs.h index ad93c57edd17..ef4746b7b5d3 100644 --- a/drivers/gpu/drm/xe/regs/xe_regs.h +++ b/drivers/gpu/drm/xe/regs/xe_regs.h @@ -7,6 +7,8 @@ #include "regs/xe_reg_defs.h" +#define SWF_SCRATCHPAD(_idx) XE_REG(0x4f000 + (_idx) * 4) + #define SOC_BASE 0x280000 #define GU_CNTL_PROTECTED XE_REG(0x10100C) diff --git a/drivers/gpu/drm/xe/tests/xe_bo.c b/drivers/gpu/drm/xe/tests/xe_bo.c index 49c95ed67d7e..6a17e13d58cf 100644 --- a/drivers/gpu/drm/xe/tests/xe_bo.c +++ b/drivers/gpu/drm/xe/tests/xe_bo.c @@ -22,6 +22,231 @@ #include "xe_pci.h" #include "xe_pm.h" +#ifdef CONFIG_DRM_XE_DEBUG_PAGE_SIZE +struct page_size_alloc_saved { + enum xe_page_size_alloc_ctrl_mode mode; + u32 cur_index; +}; + +/* Caller must hold xe->page_size_alloc_ctrl.lock. */ +static void page_size_alloc_save(struct xe_device *xe, + struct page_size_alloc_saved *s) +{ + s->mode = xe->page_size_alloc_ctrl.mode; + s->cur_index = xe->page_size_alloc_ctrl.cur_index; +} + +static void page_size_alloc_restore(struct xe_device *xe, + const struct page_size_alloc_saved *s) +{ + mutex_lock(&xe->page_size_alloc_ctrl.lock); + xe->page_size_alloc_ctrl.mode = s->mode; + xe->page_size_alloc_ctrl.cur_index = s->cur_index; + mutex_unlock(&xe->page_size_alloc_ctrl.lock); +} + +/* Expected properties for a forced page-size allocation mode. */ +struct leaf_info { + u64 leaf; + u64 alloc_size; + u32 flag; + const char *name; +}; + +static const struct leaf_info leaf_2m = { + .leaf = SZ_2M, + .alloc_size = SZ_2M - PAGE_SIZE, + .flag = XE_BO_FLAG_NEEDS_2M, + .name = "2M", +}; + +static const struct leaf_info leaf_1g = { + .leaf = SZ_1G, + .alloc_size = SZ_1G - PAGE_SIZE, + .flag = XE_BO_FLAG_NEEDS_1G, + .name = "1G", +}; + +static void run_only_leaf(struct kunit *test, + enum xe_page_size_alloc_ctrl_mode mode, + const struct leaf_info *li) +{ + struct xe_device *xe = test->priv; + struct page_size_alloc_saved saved; + struct xe_bo *bo; + struct ttm_buffer_object *ttm_bo; + u32 other_flags; + + if (!IS_DGFX(xe)) { + kunit_skip(test, "requires dGFX VRAM"); + return; + } + + mutex_lock(&xe->page_size_alloc_ctrl.lock); + page_size_alloc_save(xe, &saved); + xe->page_size_alloc_ctrl.mode = mode; + mutex_unlock(&xe->page_size_alloc_ctrl.lock); + + bo = xe_bo_create_user(xe, NULL, li->alloc_size, + DRM_XE_GEM_CPU_CACHING_WC, + XE_BO_FLAG_VRAM0, NULL); + if (IS_ERR(bo)) { + page_size_alloc_restore(xe, &saved); + if (PTR_ERR(bo) == -ENOSPC) { + kunit_skip(test, + "no contiguous %s VRAM available right now", + li->name); + return; + } + + KUNIT_FAIL(test, "%s BO alloc failed: %pe", li->name, bo); + return; + } + + ttm_bo = &bo->ttm; + + /* 1) The mode added the right NEEDS_* flag. */ + KUNIT_EXPECT_TRUE_MSG(test, bo->flags & li->flag, + "%s: flag missing, flags=0x%x", + li->name, bo->flags); + + /* 2) No other NEEDS_* flags accidentally tagged on. */ + other_flags = (XE_BO_FLAG_NEEDS_64K | + XE_BO_FLAG_NEEDS_2M | + XE_BO_FLAG_NEEDS_1G) & ~li->flag; + KUNIT_EXPECT_FALSE_MSG(test, bo->flags & other_flags, + "%s: stray flags=0x%x", + li->name, bo->flags); + /* 3) BO size was rounded up to the expected leaf size. */ + KUNIT_EXPECT_EQ_MSG(test, xe_bo_size(bo), li->leaf, + "%s: bo size=%llu expected=%llu", + li->name, + (u64)xe_bo_size(bo), + (u64)li->leaf); + /* + * 4) Allocator honored the requested alignment. + * ttm_bo->page_alignment is stored in PAGE_SIZE units, so compare against + * the expected leaf size converted with >> PAGE_SHIFT. + */ + KUNIT_EXPECT_EQ_MSG(test, ttm_bo->page_alignment, + li->leaf >> PAGE_SHIFT, + "%s: page_alignment=%u pages expected=%llu pages", + li->name, ttm_bo->page_alignment, + (u64)(li->leaf >> PAGE_SHIFT)); + + xe_bo_put(bo); + page_size_alloc_restore(xe, &saved); +} + +static void xe_bo_page_size_alloc_only_2m(struct kunit *test) +{ + run_only_leaf(test, XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_2M, &leaf_2m); +} + +static void xe_bo_page_size_alloc_only_1g(struct kunit *test) +{ + run_only_leaf(test, XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_1G, &leaf_1g); +} + +static void xe_bo_page_size_alloc_mixed_bos(struct kunit *test) +{ + struct xe_device *xe = test->priv; + struct page_size_alloc_saved saved; + struct xe_bo *bo; + struct ttm_buffer_object *ttm_bo; + u32 all_flags = XE_BO_FLAG_NEEDS_64K | XE_BO_FLAG_NEEDS_2M | + XE_BO_FLAG_NEEDS_1G; + u32 flags; + u64 expected_align; + int i; + const int n = 4; + + if (!IS_DGFX(xe)) { + kunit_skip(test, "requires dGFX VRAM"); + return; + } + + mutex_lock(&xe->page_size_alloc_ctrl.lock); + page_size_alloc_save(xe, &saved); + mutex_unlock(&xe->page_size_alloc_ctrl.lock); + + for (i = 0; i < n; i++) { + mutex_lock(&xe->page_size_alloc_ctrl.lock); + xe->page_size_alloc_ctrl.mode = XE_PAGE_SIZE_ALLOC_CTRL_MODE_MIXED; + xe->page_size_alloc_ctrl.cur_index = i; + mutex_unlock(&xe->page_size_alloc_ctrl.lock); + /* + * Request a size valid for any mixed-mode slot. Since cur_index is + * device-global and may be perturbed by concurrent allocations on + * a live system, do not assume this iteration will see a specific + * slot. + */ + bo = xe_bo_create_user(xe, NULL, SZ_1G, + DRM_XE_GEM_CPU_CACHING_WC, + XE_BO_FLAG_VRAM0, NULL); + if (IS_ERR(bo)) { + int err = PTR_ERR(bo); + + page_size_alloc_restore(xe, &saved); + if (err == -ENOSPC) { + kunit_skip(test, + "mixed mode BO allocation unavailable: %d", + err); + return; + } + KUNIT_FAIL(test, "iter=%d alloc failed: %pe", i, bo); + return; + } + + ttm_bo = &bo->ttm; + flags = bo->flags & all_flags; + /* + * Mixed mode may result in: + * 0-> default platform VRAM alignment + * XE_BO_FLAG_NEEDS_64K + * XE_BO_FLAG_NEEDS_2M + * XE_BO_FLAG_NEEDS_1G + * Any other combination is invalid. + */ + if (flags == 0) { + expected_align = SZ_4K; + if (xe->info.vram_flags & XE_VRAM_FLAGS_NEED64K) + expected_align = SZ_64K; + } else if (flags == XE_BO_FLAG_NEEDS_64K) { + expected_align = SZ_64K; + } else if (flags == XE_BO_FLAG_NEEDS_2M) { + expected_align = SZ_2M; + } else if (flags == XE_BO_FLAG_NEEDS_1G) { + expected_align = SZ_1G; + } else { + KUNIT_FAIL(test, + "iter=%d invalid mixed-mode flags: 0x%x", + i, flags); + xe_bo_put(bo); + page_size_alloc_restore(xe, &saved); + return; + } + /* + * BO size should remain valid for the selected mode. Since the + * request is SZ_1G, it should remain unchanged regardless of the + * selected page-size policy. + */ + KUNIT_EXPECT_EQ_MSG(test, xe_bo_size(bo), (u64)SZ_1G, + "iter=%d size=%llu expected=%llu", + i, + (u64)xe_bo_size(bo), + (u64)SZ_1G); + KUNIT_EXPECT_EQ_MSG(test, ttm_bo->page_alignment, + expected_align >> PAGE_SHIFT, + "iter=%d flags=0x%x page_alignment=%u pages expected=%llu pages", + i, flags, ttm_bo->page_alignment, + (u64)(expected_align >> PAGE_SHIFT)); + xe_bo_put(bo); + } + page_size_alloc_restore(xe, &saved); +} +#endif + static int ccs_test_migrate(struct xe_tile *tile, struct xe_bo *bo, bool clear, u64 get_val, u64 assign_val, struct kunit *test, struct drm_exec *exec) @@ -609,6 +834,23 @@ static struct kunit_case xe_bo_tests[] = { {} }; +#ifdef CONFIG_DRM_XE_DEBUG_PAGE_SIZE +static struct kunit_case xe_bo_page_size_alloc_cases[] = { + KUNIT_CASE_PARAM(xe_bo_page_size_alloc_only_2m, xe_pci_live_device_gen_param), + KUNIT_CASE_PARAM(xe_bo_page_size_alloc_only_1g, xe_pci_live_device_gen_param), + KUNIT_CASE_PARAM(xe_bo_page_size_alloc_mixed_bos, xe_pci_live_device_gen_param), + {} +}; + +VISIBLE_IF_KUNIT +struct kunit_suite xe_bo_page_size_alloc_suite = { + .name = "xe_bo_page_size_alloc", + .test_cases = xe_bo_page_size_alloc_cases, + .init = xe_kunit_helper_xe_device_live_test_init, +}; +EXPORT_SYMBOL_IF_KUNIT(xe_bo_page_size_alloc_suite); +#endif + VISIBLE_IF_KUNIT struct kunit_suite xe_bo_test_suite = { .name = "xe_bo", diff --git a/drivers/gpu/drm/xe/tests/xe_guc_klv_helpers_kunit.c b/drivers/gpu/drm/xe/tests/xe_guc_klv_helpers_kunit.c new file mode 100644 index 000000000000..82869363ab7e --- /dev/null +++ b/drivers/gpu/drm/xe/tests/xe_guc_klv_helpers_kunit.c @@ -0,0 +1,429 @@ +// SPDX-License-Identifier: GPL-2.0 AND MIT +/* + * Copyright © 2026 Intel Corporation + */ + +#include <kunit/test.h> +#include <kunit/test-bug.h> + +#define TEST_KEY (GUC_KLV_RESERVED_RANGE_START + 0x3de) +#define TEST_GROUP_KEY (GUC_KLV_RESERVED_RANGE_START + 0x3f0) +#define TEST_PAD 0xdeadbeef + +static bool fake_is_group_key(u16 key) +{ + return is_reserved_key(key) && key >= TEST_GROUP_KEY; +} + +static void test_count(struct kunit *test) +{ + u32 value = 0x12345678; + u16 key = TEST_KEY; + u32 klvs[] = { + PREP_GUC_KLV(key + 0, 0), + PREP_GUC_KLV(key + 1, 1), value, + PREP_GUC_KLV(key + 2, 2), value, value, + PREP_GUC_KLV(key + 3, 0), + 0, /* padding */ + }; + + KUNIT_EXPECT_EQ(test, 0, xe_guc_klv_count(klvs, 0)); + KUNIT_EXPECT_EQ(test, 1, xe_guc_klv_count(klvs, 1)); + KUNIT_EXPECT_GT(test, 0, xe_guc_klv_count(klvs, 2)); + KUNIT_EXPECT_EQ(test, 2, xe_guc_klv_count(klvs, 3)); + KUNIT_EXPECT_GT(test, 0, xe_guc_klv_count(klvs, 4)); + KUNIT_EXPECT_GT(test, 0, xe_guc_klv_count(klvs, 5)); + KUNIT_EXPECT_EQ(test, 3, xe_guc_klv_count(klvs, 6)); + KUNIT_EXPECT_EQ(test, 4, xe_guc_klv_count(klvs, 7)); + + /* 0 is treated as reserved KLV { KEY=0, LEN=0 } */ + KUNIT_EXPECT_EQ(test, 5, xe_guc_klv_count(klvs, 8)); +} + +static void test_encode_u32(struct kunit *test) +{ + u32 *fail = ERR_PTR(-ENOMEM); + u32 value = 0x12345678; + u16 key = TEST_KEY; + u32 klvs[16]; + + memset32(klvs, TEST_PAD, ARRAY_SIZE(klvs)); + + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-ENOSPC), xe_guc_klv_encode_u32(klvs, 0, key, value)); + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-ENOSPC), xe_guc_klv_encode_u32(klvs, 1, key, value)); + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, xe_guc_klv_encode_u32(klvs, 2, key, value)); + KUNIT_EXPECT_EQ(test, klvs[0], PREP_GUC_KLV(key, 1)); + KUNIT_EXPECT_EQ(test, klvs[1], value); + KUNIT_EXPECT_EQ(test, klvs[2], TEST_PAD); + KUNIT_EXPECT_PTR_EQ(test, &klvs[2], xe_guc_klv_encode_u32(klvs, 2, key, value)); + KUNIT_EXPECT_PTR_EQ(test, + xe_guc_klv_encode_u32(klvs, 2, key, value), + xe_guc_klv_encode_u32(klvs, ARRAY_SIZE(klvs), key, value)); + + KUNIT_ASSERT_PTR_EQ(test, fail, xe_guc_klv_encode_u32(fail, ARRAY_SIZE(klvs), key, value)); +} + +static void test_encode_u64(struct kunit *test) +{ + u64 value = 0x123456789abcdef0; + u32 *fail = ERR_PTR(-ENOMEM); + u16 key = TEST_KEY; + u32 klvs[16]; + + memset32(klvs, TEST_PAD, ARRAY_SIZE(klvs)); + + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-ENOSPC), xe_guc_klv_encode_u64(klvs, 0, key, value)); + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-ENOSPC), xe_guc_klv_encode_u64(klvs, 1, key, value)); + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-ENOSPC), xe_guc_klv_encode_u64(klvs, 2, key, value)); + + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, xe_guc_klv_encode_u64(klvs, 3, key, value)); + KUNIT_EXPECT_EQ(test, klvs[0], PREP_GUC_KLV(key, 2)); + KUNIT_EXPECT_EQ(test, klvs[1], lower_32_bits(value)); + KUNIT_EXPECT_EQ(test, klvs[2], upper_32_bits(value)); + KUNIT_EXPECT_EQ(test, klvs[3], TEST_PAD); + KUNIT_EXPECT_PTR_EQ(test, &klvs[3], xe_guc_klv_encode_u64(klvs, 3, key, value)); + KUNIT_EXPECT_PTR_EQ(test, + xe_guc_klv_encode_u64(klvs, 3, key, value), + xe_guc_klv_encode_u64(klvs, ARRAY_SIZE(klvs), key, value)); + + KUNIT_ASSERT_PTR_EQ(test, fail, xe_guc_klv_encode_u64(fail, ARRAY_SIZE(klvs), key, value)); +} + +static u32 str_klv_size(const char *string) +{ + return GUC_KLV_LEN_MIN + to_num_dwords(strlen(string) + 1); +} + +static void test_encode_string(struct kunit *test) +{ + size_t longest_str = to_num_bytes(FIELD_MAX(GUC_KLV_0_LEN)) - 1; + u32 avail = GUC_KLV_LEN_MIN + FIELD_MAX(GUC_KLV_0_LEN) + 1; + const char *string = "abcdefghijklmnopqrstvwxyz"; + u16 key = TEST_KEY; + u32 *klvs; + u32 *next; + char *buf; + u32 n; + + klvs = kunit_kcalloc(test, avail, sizeof(u32), GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, klvs); + + buf = kunit_kzalloc(test, longest_str + 2, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); + + /* empty string, no space, must fail */ + for (n = 0; n < str_klv_size(""); n++) { + klvs[0] = TEST_PAD; + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-ENOSPC), + xe_guc_klv_encode_string(klvs, n, key, "")); + KUNIT_EXPECT_EQ(test, klvs[0], TEST_PAD); + } + + /* empty string, must pass */ + KUNIT_EXPECT_PTR_EQ(test, klvs + str_klv_size(""), + xe_guc_klv_encode_string(klvs, str_klv_size(""), key, "")); + KUNIT_EXPECT_PTR_EQ(test, klvs + str_klv_size(""), + xe_guc_klv_encode_string(klvs, avail, key, "")); + + /* demo string, no space, must fail */ + for (n = 0; n < str_klv_size(string); n++) { + klvs[0] = TEST_PAD; + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-ENOSPC), + xe_guc_klv_encode_string(klvs, n, key, string)); + KUNIT_EXPECT_EQ(test, klvs[0], TEST_PAD); + } + + /* different string len, must pass */ + for (n = 0; n <= strlen(string); n++) { + strscpy(buf, string, n + 1); + kunit_info(test, "%u: '%s'\n", n, buf); + KUNIT_ASSERT_EQ(test, n, strlen(buf)); + memset32(klvs, TEST_PAD, avail); + + next = xe_guc_klv_encode_string(klvs, str_klv_size(buf), key, buf); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, next); + KUNIT_EXPECT_PTR_EQ(test, next, klvs + str_klv_size(buf)); + KUNIT_EXPECT_STREQ_MSG(test, buf, (char *)(klvs + GUC_KLV_LEN_MIN), "n=%u", n); + kunit_info(test, "%u: %*ph\n", n, (int)to_num_bytes(next - klvs), klvs); + KUNIT_EXPECT_NE(test, *(next - 1), TEST_PAD); + KUNIT_ASSERT_EQ(test, *next, TEST_PAD); + + /* bigger buf doesn't matter */ + KUNIT_EXPECT_PTR_EQ(test, + xe_guc_klv_encode_string(klvs, str_klv_size(buf), key, buf), + xe_guc_klv_encode_string(klvs, avail, key, buf)); + } + + /* don't crash if already failed */ + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-EROFS), + xe_guc_klv_encode_string(ERR_PTR(-EROFS), avail, key, "")); + + /* too long string, must fail */ + memset(buf, 'X', longest_str + 1); + buf[longest_str + 1] = '\0'; + KUNIT_EXPECT_LT(test, longest_str, strlen(buf)); + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-E2BIG), + xe_guc_klv_encode_string(klvs, avail, key, buf)); + + /* longest string, should pass */ + buf[longest_str] = '\0'; + KUNIT_EXPECT_EQ(test, longest_str, strlen(buf)); + KUNIT_EXPECT_PTR_EQ(test, klvs + str_klv_size(buf), + xe_guc_klv_encode_string(klvs, avail, key, buf)); +} + +struct some_object { + u32 value1; + u64 value2; +} __packed; + +static u32 *obj_raw_encoder(u32 *klvs, u32 avail, const void *arg) +{ + const struct some_object *obj = arg; + size_t sz = sizeof(*obj); + u32 dwords = to_num_dwords(sz); + + if (IS_ERR(klvs)) + return klvs; + if (dwords > avail) + return ERR_PTR(-ENOSPC); + memcpy(klvs, obj, sz); + return klvs + dwords; +} + +static u32 *obj_klv_encoder(u32 *klvs, u32 avail, const void *arg) +{ + const struct some_object *obj = arg; + u32 *end = klvs + avail; + + klvs = xe_guc_klv_encode_u32(klvs, end - klvs, TEST_KEY + 1, obj->value1); + klvs = xe_guc_klv_encode_u64(klvs, end - klvs, TEST_KEY + 2, obj->value2); + return klvs; +} + +static u32 *obj_nested_encoder(u32 *klvs, u32 avail, const void *arg) +{ + u32 *end = klvs + avail; + + klvs = xe_guc_klv_encode_object(klvs, end - klvs, TEST_GROUP_KEY + 1, + arg, obj_klv_encoder); + klvs = xe_guc_klv_encode_object(klvs, end - klvs, TEST_GROUP_KEY + 2, + arg, obj_klv_encoder); + return klvs; +} + +static void test_encode_object_raw(struct kunit *test) +{ + const struct some_object obj = { + .value1 = 0xdead1234, + .value2 = 0xdead87654321dead, + }; + u32 payload = to_num_dwords(sizeof(obj)); + u32 *err = ERR_PTR(-ENOSPC); + u16 key = TEST_KEY; + u32 klvs[16]; + u32 n; + + /* too small, must fail */ + for (n = 0; n < GUC_KLV_LEN_MIN + payload; n++) { + memset32(klvs, TEST_PAD, ARRAY_SIZE(klvs)); + KUNIT_EXPECT_PTR_EQ_MSG(test, ERR_PTR(-ENOSPC), + xe_guc_klv_encode_object(klvs, n, key, &obj, + obj_raw_encoder), + "buf size=%u dwords", n); + } + + /* must pass */ + memset32(klvs, TEST_PAD, ARRAY_SIZE(klvs)); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, + xe_guc_klv_encode_object(klvs, GUC_KLV_LEN_MIN + payload, + key, &obj, obj_raw_encoder)); + KUNIT_EXPECT_EQ(test, klvs[0], PREP_GUC_KLV(key, payload)); + KUNIT_EXPECT_MEMEQ(test, &klvs[1], &obj, sizeof(obj)); + + /* already failed, must fail */ + KUNIT_ASSERT_PTR_EQ(test, err, + xe_guc_klv_encode_object(err, ARRAY_SIZE(klvs), key, + &obj, obj_raw_encoder)); +} + +static void test_encode_object_klv(struct kunit *test) +{ + const struct some_object obj = { + .value1 = 0xdead1234, + .value2 = 0xdead87654321dead, + }; + u16 key = TEST_GROUP_KEY; + u32 payload = 0; + u32 klvs[16]; + u32 n; + + payload += GUC_KLV_LEN_MIN + to_num_dwords(sizeof(obj.value1)); + payload += GUC_KLV_LEN_MIN + to_num_dwords(sizeof(obj.value2)); + + /* too small, must fail */ + for (n = 0; n < GUC_KLV_LEN_MIN + payload; n++) { + memset32(klvs, TEST_PAD, ARRAY_SIZE(klvs)); + KUNIT_EXPECT_PTR_EQ_MSG(test, ERR_PTR(-ENOSPC), + xe_guc_klv_encode_object(klvs, n, key, &obj, + obj_klv_encoder), + "buf size=%u dwords", n); + } + + /* must pass */ + memset32(klvs, TEST_PAD, ARRAY_SIZE(klvs)); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, + xe_guc_klv_encode_object(klvs, GUC_KLV_LEN_MIN + payload, + key, &obj, obj_klv_encoder)); + KUNIT_EXPECT_EQ(test, klvs[0], PREP_GUC_KLV(key, payload)); + KUNIT_EXPECT_EQ(test, klvs[1], PREP_GUC_KLV(TEST_KEY + 1, 1)); + KUNIT_EXPECT_EQ(test, klvs[2], obj.value1); + KUNIT_EXPECT_EQ(test, klvs[3], PREP_GUC_KLV(TEST_KEY + 2, 2)); + KUNIT_EXPECT_EQ(test, klvs[4], lower_32_bits(obj.value2)); + KUNIT_EXPECT_EQ(test, klvs[5], upper_32_bits(obj.value2)); + KUNIT_EXPECT_EQ(test, klvs[6], TEST_PAD); +} + +static void test_encode_object_nested(struct kunit *test) +{ + const struct some_object obj = { + .value1 = 0xdead1234, + .value2 = 0xdead87654321dead, + }; + u16 key = TEST_GROUP_KEY; + u32 payload = 0; + u32 klvs[16]; + u32 n; + + payload += GUC_KLV_LEN_MIN; + payload += GUC_KLV_LEN_MIN + to_num_dwords(sizeof(obj.value1)); + payload += GUC_KLV_LEN_MIN + to_num_dwords(sizeof(obj.value2)); + payload *= 2; + + /* too small, must fail */ + for (n = 0; n < GUC_KLV_LEN_MIN + payload; n++) { + memset32(klvs, TEST_PAD, ARRAY_SIZE(klvs)); + KUNIT_EXPECT_PTR_EQ_MSG(test, ERR_PTR(-ENOSPC), + xe_guc_klv_encode_object(klvs, n, key, &obj, + obj_nested_encoder), + "buf size=%u dwords", n); + } + + /* must pass */ + memset32(klvs, TEST_PAD, ARRAY_SIZE(klvs)); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, + xe_guc_klv_encode_object(klvs, GUC_KLV_LEN_MIN + payload, + key, &obj, obj_nested_encoder)); + KUNIT_EXPECT_EQ(test, klvs[0], PREP_GUC_KLV(key, payload)); + KUNIT_EXPECT_EQ(test, klvs[1], PREP_GUC_KLV(TEST_GROUP_KEY + 1, 5)); + KUNIT_EXPECT_EQ(test, klvs[2], PREP_GUC_KLV(TEST_KEY + 1, 1)); + KUNIT_EXPECT_EQ(test, klvs[3], obj.value1); + KUNIT_EXPECT_EQ(test, klvs[4], PREP_GUC_KLV(TEST_KEY + 2, 2)); + KUNIT_EXPECT_EQ(test, klvs[5], lower_32_bits(obj.value2)); + KUNIT_EXPECT_EQ(test, klvs[6], upper_32_bits(obj.value2)); + KUNIT_EXPECT_EQ(test, klvs[7], PREP_GUC_KLV(TEST_GROUP_KEY + 2, 5)); + KUNIT_EXPECT_EQ(test, klvs[8], PREP_GUC_KLV(TEST_KEY + 1, 1)); + KUNIT_EXPECT_EQ(test, klvs[9], obj.value1); + KUNIT_EXPECT_EQ(test, klvs[10], PREP_GUC_KLV(TEST_KEY + 2, 2)); + KUNIT_EXPECT_EQ(test, klvs[11], lower_32_bits(obj.value2)); + KUNIT_EXPECT_EQ(test, klvs[12], upper_32_bits(obj.value2)); + KUNIT_EXPECT_EQ(test, klvs[13], TEST_PAD); +} + +static u32 *obj_echo_encoder(u32 *klvs, u32 avail, const void *arg) +{ + return ERR_CAST(arg); +} + +static void test_encode_object_basic(struct kunit *test) +{ + u32 longest = GUC_KLV_LEN_MIN + FIELD_MAX(GUC_KLV_0_LEN); + u32 avail = GUC_KLV_LEN_MIN + longest; + u16 key = TEST_GROUP_KEY; + u32 *klvs; + + klvs = kunit_kcalloc(test, avail, sizeof(u32), GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, klvs); + + /* smallest */ + KUNIT_EXPECT_PTR_EQ(test, klvs + GUC_KLV_LEN_MIN, + xe_guc_klv_encode_object(klvs, avail, key, + klvs + GUC_KLV_LEN_MIN, + obj_echo_encoder)); + /* largest */ + KUNIT_EXPECT_PTR_EQ(test, klvs + longest, + xe_guc_klv_encode_object(klvs, avail, key, + klvs + longest, + obj_echo_encoder)); + /* already failed */ + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-EROFS), + xe_guc_klv_encode_object(ERR_PTR(-EROFS), avail, key, + klvs + GUC_KLV_LEN_MIN, + obj_echo_encoder)); + /* encoding error */ + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-EUCLEAN), + xe_guc_klv_encode_object(klvs, avail, key, + ERR_PTR(-EUCLEAN), + obj_echo_encoder)); + /* no space */ + KUNIT_EXPECT_PTR_EQ(test, ERR_PTR(-ENOSPC), + xe_guc_klv_encode_object(klvs, 0, key, + klvs + GUC_KLV_LEN_MIN, + obj_echo_encoder)); +} + +static void __drm_printfn_kunit(struct drm_printer *p, struct va_format *vaf) +{ + struct kunit *test = p->arg; + + kunit_info(test, "%pV", vaf); +} + +static struct drm_printer drm_kunit_printer(void) +{ + struct drm_printer p = { + .printfn = __drm_printfn_kunit, + .arg = kunit_get_current_test(), + }; + return p; +} + +static void test_print(struct kunit *test) +{ + struct drm_printer p = drm_kunit_printer(); + u32 zeros[] = { 0, 0, 0, /* padding */ }; + u32 klvs[] = { + PREP_GUC_KLV(GUC_KLV_OPT_IN_FEATURE_EXT_CAT_ERR_TYPE_KEY, 0), + PREP_GUC_KLV(GUC_KLV_VF_CFG_NUM_CONTEXTS_KEY, 1), 1234, + PREP_GUC_KLV(GUC_KLV_VF_CFG_GGTT_SIZE_KEY, 2), 0x4000, 0x0123, + PREP_GUC_KLV(TEST_KEY, 3), 1, 2, 3, + PREP_GUC_KLV(TEST_GROUP_KEY, 5), + PREP_GUC_KLV(TEST_KEY + 1, 1), 1, + PREP_GUC_KLV(TEST_KEY + 2, 2), 1, 2, + }; + + kunit_activate_static_stub(test, is_group_key, fake_is_group_key); + xe_guc_klv_print(zeros, ARRAY_SIZE(zeros), &p); + xe_guc_klv_print(klvs, ARRAY_SIZE(klvs), &p); +} + +static struct kunit_case guc_klv_helpers_test_cases[] = { + KUNIT_CASE(test_count), + KUNIT_CASE(test_encode_u32), + KUNIT_CASE(test_encode_u64), + KUNIT_CASE(test_encode_string), + KUNIT_CASE(test_encode_object_raw), + KUNIT_CASE(test_encode_object_klv), + KUNIT_CASE(test_encode_object_nested), + KUNIT_CASE(test_encode_object_basic), + KUNIT_CASE(test_print), + {} +}; + +static struct kunit_suite guc_klv_helpers_suite = { + .name = "guc_klv_helpers", + .test_cases = guc_klv_helpers_test_cases, +}; + +kunit_test_suite(guc_klv_helpers_suite); diff --git a/drivers/gpu/drm/xe/tests/xe_live_test_mod.c b/drivers/gpu/drm/xe/tests/xe_live_test_mod.c index c55e46f1ae92..87cd7db20e5f 100644 --- a/drivers/gpu/drm/xe/tests/xe_live_test_mod.c +++ b/drivers/gpu/drm/xe/tests/xe_live_test_mod.c @@ -11,6 +11,9 @@ extern struct kunit_suite xe_dma_buf_test_suite; extern struct kunit_suite xe_migrate_test_suite; extern struct kunit_suite xe_mocs_test_suite; extern struct kunit_suite xe_guc_g2g_test_suite; +#ifdef CONFIG_DRM_XE_DEBUG_PAGE_SIZE +extern struct kunit_suite xe_bo_page_size_alloc_suite; +#endif kunit_test_suite(xe_bo_test_suite); kunit_test_suite(xe_bo_shrink_test_suite); @@ -18,6 +21,9 @@ kunit_test_suite(xe_dma_buf_test_suite); kunit_test_suite(xe_migrate_test_suite); kunit_test_suite(xe_mocs_test_suite); kunit_test_suite(xe_guc_g2g_test_suite); +#ifdef CONFIG_DRM_XE_DEBUG_PAGE_SIZE +kunit_test_suite(xe_bo_page_size_alloc_suite); +#endif MODULE_AUTHOR("Intel Corporation"); MODULE_LICENSE("GPL"); diff --git a/drivers/gpu/drm/xe/tests/xe_pci.c b/drivers/gpu/drm/xe/tests/xe_pci.c index 8df9029afcd3..bb0393475524 100644 --- a/drivers/gpu/drm/xe/tests/xe_pci.c +++ b/drivers/gpu/drm/xe/tests/xe_pci.c @@ -9,7 +9,6 @@ #include <kunit/test-bug.h> #include <kunit/test.h> -#include <kunit/test-bug.h> #include <kunit/visibility.h> #define PLATFORM_CASE(platform__, graphics_step__) \ diff --git a/drivers/gpu/drm/xe/tests/xe_sriov_packet_kunit.c b/drivers/gpu/drm/xe/tests/xe_sriov_packet_kunit.c new file mode 100644 index 000000000000..c2720b461dee --- /dev/null +++ b/drivers/gpu/drm/xe/tests/xe_sriov_packet_kunit.c @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-2.0 AND MIT +/* + * Copyright © 2026 Intel Corporation + */ + +#include <kunit/test.h> +#include <kunit/test-bug.h> + +#include "xe_device.h" +#include "xe_guc_klv_helpers.h" +#include "xe_kunit_helpers.h" +#include "xe_pci_test.h" + +#define TEST_VF VFID(1) + +static int sriov_packet_test_init(struct kunit *test) +{ + struct xe_pci_fake_data fake = { + .sriov_mode = XE_SRIOV_MODE_PF, + .platform = XE_PANTHERLAKE, /* we need MEMIRQ */ + .subplatform = XE_SUBPLATFORM_NONE, + .graphics_verx100 = 3000, + .media_verx100 = 3000, + }; + struct xe_device *xe; + + test->priv = &fake; + xe_kunit_helper_xe_device_test_init(test); + xe = test->priv; + + /* pretend we can support at least VF1 */ + xe->sriov.pf.device_total_vfs = 1; + xe->sriov.pf.driver_max_vfs = 1; + + KUNIT_ASSERT_EQ(test, 0, xe_sriov_init(xe)); + KUNIT_ASSERT_TRUE(test, xe_sriov_pf_migration_supported(xe)); + + return 0; +} + +static void test_descriptor_init(struct kunit *test) +{ + struct xe_device *xe = test->priv; + struct xe_sriov_packet **desc; + + /* note: with lock held we should avoid KUNIT_ASSERT() */ + guard(mutex)(pf_migration_mutex(xe, TEST_VF)); + + KUNIT_EXPECT_EQ(test, 0, pf_descriptor_init(xe, TEST_VF)); + desc = pf_pick_descriptor(xe, TEST_VF); + KUNIT_EXPECT_NOT_ERR_OR_NULL(test, *desc); + if (!*desc) + return; + KUNIT_EXPECT_NE(test, (*desc)->hdr.version, 0); + KUNIT_EXPECT_EQ(test, (*desc)->hdr.version, XE_SRIOV_PACKET_SUPPORTED_VERSION); + KUNIT_EXPECT_EQ(test, (*desc)->hdr.type, XE_SRIOV_PACKET_TYPE_DESCRIPTOR); + KUNIT_EXPECT_NE(test, (*desc)->hdr.size, 0); + KUNIT_EXPECT_NOT_ERR_OR_NULL(test, (*desc)->vaddr); + if (!(*desc)->vaddr) + return; + KUNIT_EXPECT_EQ(test, 0, xe_sriov_packet_process_descriptor(xe, TEST_VF, *desc)); + + switch ((*desc)->hdr.version) { + case 1: + /* v1 is KLV based */ + KUNIT_EXPECT_TRUE(test, IS_ALIGNED((*desc)->hdr.size, sizeof(u32))); + /* v1 has at least DEVID and REVID KLVs */ + KUNIT_EXPECT_LE(test, 2, + xe_guc_klv_count((*desc)->vaddr, + (*desc)->hdr.size / sizeof(u32))); + break; + default: + kunit_mark_skipped(test, "no test code for version %u\n", (*desc)->hdr.version); + return; + } +} + +static struct kunit_case sriov_packet_test_cases[] = { + KUNIT_CASE(test_descriptor_init), + {} +}; + +static struct kunit_suite sriov_packet_suite = { + .name = "sriov_packet", + .test_cases = sriov_packet_test_cases, + .init = sriov_packet_test_init, +}; + +kunit_test_suite(sriov_packet_suite); diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c index 85e6d9a0f575..dde309821237 100644 --- a/drivers/gpu/drm/xe/xe_bo.c +++ b/drivers/gpu/drm/xe/xe_bo.c @@ -341,6 +341,18 @@ static void xe_evict_flags(struct ttm_buffer_object *tbo, return; } + if (xe_bo_madv_is_dontneed(bo)) { + /* + * We can't use purge_placement here, since we need to trigger + * our own purge procedure at the start of xe_bo_move(), which + * would otherwise be skipped. At the same time we don't want + * ttm to then populate the tt with dst pages, before the move + * callback, hence use sys_placement here. + */ + *placement = sys_placement; + return; + } + /* * For xe, sg bos that are evicted to system just triggers a * rebind of the sg list upon subsequent validation to XE_PL_TT. @@ -1102,6 +1114,21 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, xe_pm_runtime_get_noresume(xe); } + /* + * Attach CCS BBs before submitting the copy job below so a VF + * migration racing the copy sees valid, up to date attach state. + */ + if (IS_VF_CCS_READY(xe) && + ((move_lacks_source && new_mem->mem_type == XE_PL_TT) || + (old_mem_type == XE_PL_SYSTEM && new_mem->mem_type == XE_PL_TT)) && + handle_system_ccs) { + ret = xe_sriov_vf_ccs_attach_bo(bo, new_mem); + if (ret) { + xe_pm_runtime_put(xe); + goto out; + } + } + if (move_lacks_source) { u32 flags = 0; @@ -1139,22 +1166,19 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, ttm_bo_move_null(ttm_bo, new_mem); } - dma_fence_put(fence); - xe_pm_runtime_put(xe); - /* - * CCS meta data is migrated from TT -> SMEM. So, let us detach the - * BBs from BO as it is no longer needed. + * Detach must wait for the copy above to complete: a VF migration + * racing an in-flight copy must still see valid CCS BBs, so don't + * tear them down until the copy fence has signaled. */ if (IS_VF_CCS_READY(xe) && old_mem_type == XE_PL_TT && - new_mem->mem_type == XE_PL_SYSTEM) + new_mem->mem_type == XE_PL_SYSTEM) { + dma_fence_wait(fence, false); xe_sriov_vf_ccs_detach_bo(bo); + } - if (IS_VF_CCS_READY(xe) && - ((move_lacks_source && new_mem->mem_type == XE_PL_TT) || - (old_mem_type == XE_PL_SYSTEM && new_mem->mem_type == XE_PL_TT)) && - handle_system_ccs) - ret = xe_sriov_vf_ccs_attach_bo(bo); + dma_fence_put(fence); + xe_pm_runtime_put(xe); out: if ((!ttm_bo->resource || ttm_bo->resource->mem_type == XE_PL_SYSTEM) && @@ -1349,7 +1373,7 @@ int xe_bo_notifier_prepare_pinned(struct xe_bo *bo) backup = xe_bo_init_locked(xe, NULL, NULL, bo->ttm.base.resv, NULL, xe_bo_size(bo), DRM_XE_GEM_CPU_CACHING_WB, ttm_bo_type_kernel, XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS | - XE_BO_FLAG_PINNED, &exec); + XE_BO_FLAG_PINNED, NULL, &exec); if (IS_ERR(backup)) { drm_exec_retry_on_contention(&exec); ret = PTR_ERR(backup); @@ -1490,7 +1514,7 @@ int xe_bo_evict_pinned(struct xe_bo *bo) xe_bo_size(bo), DRM_XE_GEM_CPU_CACHING_WB, ttm_bo_type_kernel, XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS | - XE_BO_FLAG_PINNED, &exec); + XE_BO_FLAG_PINNED, NULL, &exec); if (IS_ERR(backup)) { drm_exec_retry_on_contention(&exec); ret = PTR_ERR(backup); @@ -1826,6 +1850,8 @@ static void xe_ttm_bo_destroy(struct ttm_buffer_object *ttm_bo) if (bo->ttm.base.import_attach) drm_prime_gem_destroy(&bo->ttm.base, NULL); + if (bo->dma_buf) + dma_buf_put(bo->dma_buf); drm_gem_object_release(&bo->ttm.base); xe_assert(xe, list_empty(&ttm_bo->base.gpuva.list)); @@ -2283,6 +2309,8 @@ void xe_bo_free(struct xe_bo *bo) * @cpu_caching: The cpu caching used for system memory backing store. * @type: The TTM buffer object type. * @flags: XE_BO_FLAG_ flags. + * @dma_buf: The dma-buf to reference for the BO lifetime (imported BOs), + * or NULL. * @exec: The drm_exec transaction to use for exhaustive eviction. * * Initialize or create an xe buffer object. On failure, any allocated buffer @@ -2294,7 +2322,8 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo, struct xe_tile *tile, struct dma_resv *resv, struct ttm_lru_bulk_move *bulk, size_t size, u16 cpu_caching, enum ttm_bo_type type, - u32 flags, struct drm_exec *exec) + u32 flags, struct dma_buf *dma_buf, + struct drm_exec *exec) { struct ttm_operation_ctx ctx = { .interruptible = true, @@ -2323,8 +2352,16 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo, if (flags & (XE_BO_FLAG_VRAM_MASK | XE_BO_FLAG_STOLEN) && !(flags & XE_BO_FLAG_IGNORE_MIN_PAGE_SIZE) && ((xe->info.vram_flags & XE_VRAM_FLAGS_NEED64K) || - (flags & (XE_BO_FLAG_NEEDS_64K | XE_BO_FLAG_NEEDS_2M)))) { - size_t align = flags & XE_BO_FLAG_NEEDS_2M ? SZ_2M : SZ_64K; + (flags & (XE_BO_FLAG_NEEDS_64K | XE_BO_FLAG_NEEDS_2M | + XE_BO_FLAG_NEEDS_1G)))) { + size_t align; + + if (flags & XE_BO_FLAG_NEEDS_1G) + align = SZ_1G; + else if (flags & XE_BO_FLAG_NEEDS_2M) + align = SZ_2M; + else + align = SZ_64K; aligned_size = ALIGN(size, align); if (type != ttm_bo_type_device) @@ -2383,6 +2420,17 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo, placement = (type == ttm_bo_type_sg || bo->flags & XE_BO_FLAG_DEFER_BACKING) ? &sys_placement : &bo->placement; + + /* + * For imported BOs, keep the exporter dma-buf alive for the BO + * lifetime. Taken before ttm_bo_init_reserved() to also cover a + * creation failure there. Released in xe_ttm_bo_destroy(). + */ + if (dma_buf) { + get_dma_buf(dma_buf); + bo->dma_buf = dma_buf; + } + err = ttm_bo_init_reserved(&xe->ttm, &bo->ttm, type, placement, alignment, &ctx, NULL, resv, xe_ttm_bo_destroy); @@ -2500,7 +2548,7 @@ __xe_bo_create_locked(struct xe_device *xe, vm && !xe_vm_in_fault_mode(vm) && flags & XE_BO_FLAG_USER ? &vm->lru_bulk_move : NULL, size, - cpu_caching, type, flags, exec); + cpu_caching, type, flags, NULL, exec); if (IS_ERR(bo)) return bo; @@ -2604,6 +2652,145 @@ static struct xe_bo *xe_bo_create_novm(struct xe_device *xe, struct xe_tile *til return ret ? ERR_PTR(ret) : bo; } +#ifdef CONFIG_DRM_XE_DEBUG_PAGE_SIZE +static void xe_bo_debug_mixed_mode_cur_index_advance(struct xe_device *xe, struct xe_bo *bo) +{ + if (!xe_debug_page_size_mode_is_mixed(xe)) + return; + + if (!(bo->flags & XE_BO_FLAG_VRAM_MASK) || + !(bo->flags & XE_BO_FLAG_USER)) + return; + + mutex_lock(&xe->page_size_alloc_ctrl.lock); + if (xe->page_size_alloc_ctrl.mode == XE_PAGE_SIZE_ALLOC_CTRL_MODE_MIXED) + xe->page_size_alloc_ctrl.cur_index++; + mutex_unlock(&xe->page_size_alloc_ctrl.lock); +} + +static bool xe_size_align_overflows(size_t size, size_t align) +{ + return size > SIZE_MAX - (align - 1); +} + +static u32 get_flag_from_cur_index_in_mixed_mode(struct xe_device *xe, size_t *align_size, + int *err) +{ + static const struct { + u32 flag; + size_t align; + } map[] = { + { 0, SZ_4K }, /* default: 4K, no flag */ + { XE_BO_FLAG_NEEDS_64K, SZ_64K }, + { XE_BO_FLAG_NEEDS_2M, SZ_2M }, + { XE_BO_FLAG_NEEDS_1G, SZ_1G }, + }; + u32 idx; + const typeof(*map) *entry; + + lockdep_assert_held(&xe->page_size_alloc_ctrl.lock); + + *err = 0; + idx = xe->page_size_alloc_ctrl.cur_index % ARRAY_SIZE(map); + + entry = &map[idx]; + + if (!entry->flag) + return 0; + + if (xe_size_align_overflows(*align_size, entry->align)) { + *err = -EINVAL; + return 0; + } + *align_size = ALIGN(*align_size, entry->align); + + return entry->flag; +} + +static int xe_bo_apply_debug_page_size_policy(struct xe_device *xe, + u32 *bo_flags, + size_t *size) +{ + enum xe_page_size_alloc_ctrl_mode mode; + u32 want = 0; + size_t align_size = *size; + int err = 0; + + /* + * The debug page-size policy is only meaningful for BOs placed in + * VRAM, where the downstream BO init path can + * actually honor the corresponding minimum page-size requirement. + */ + if (!(*bo_flags & XE_BO_FLAG_VRAM_MASK)) + return 0; + + /* + * Do not override existing page-size requirement flags, since they + * may reflect functional requirements for specific BO types. + */ + if (*bo_flags & (XE_BO_FLAG_NEEDS_64K | + XE_BO_FLAG_NEEDS_2M | + XE_BO_FLAG_NEEDS_1G)) + return 0; + + if (!READ_ONCE(xe->page_size_alloc_ctrl.mode)) + return 0; + + mutex_lock(&xe->page_size_alloc_ctrl.lock); + + mode = xe->page_size_alloc_ctrl.mode; + if (mode == XE_PAGE_SIZE_ALLOC_CTRL_MODE_NONE) { + goto out_unlock; + } else if (mode == XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_2M) { + if (xe_size_align_overflows(align_size, SZ_2M)) { + err = -EINVAL; + goto out_unlock; + } + want = XE_BO_FLAG_NEEDS_2M; + align_size = ALIGN(align_size, SZ_2M); + } else if (mode == XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_1G) { + if (xe_size_align_overflows(align_size, SZ_1G)) { + err = -EINVAL; + goto out_unlock; + } + want = XE_BO_FLAG_NEEDS_1G; + align_size = ALIGN(align_size, SZ_1G); + } else if (mode == XE_PAGE_SIZE_ALLOC_CTRL_MODE_MIXED) { + want = get_flag_from_cur_index_in_mixed_mode(xe, &align_size, &err); + if (err) + goto out_unlock; + } else { + goto out_unlock; + } + + mutex_unlock(&xe->page_size_alloc_ctrl.lock); + + *bo_flags |= want; + /* + * Apply the debug page-size policy by rounding the user BO size up to + * the selected granularity. + */ + *size = align_size; + return err; + +out_unlock: + mutex_unlock(&xe->page_size_alloc_ctrl.lock); + return err; +} +#else +static int xe_bo_apply_debug_page_size_policy(struct xe_device *xe, + u32 *bo_flags, + size_t *size) +{ + return 0; +} + +static void xe_bo_debug_mixed_mode_cur_index_advance(struct xe_device *xe, + struct xe_bo *bo) +{ +} +#endif + /** * xe_bo_create_user() - Create a user BO * @xe: The xe device. @@ -2624,9 +2811,16 @@ struct xe_bo *xe_bo_create_user(struct xe_device *xe, u32 flags, struct drm_exec *exec) { struct xe_bo *bo; + int err = 0; flags |= XE_BO_FLAG_USER; + if (xe_debug_page_size_mode_not_none(xe)) { + err = xe_bo_apply_debug_page_size_policy(xe, &flags, &size); + if (err) + return ERR_PTR(err); + } + if (vm || exec) { xe_assert(xe, exec); bo = __xe_bo_create_locked(xe, NULL, vm, size, 0, ~0ULL, @@ -3441,6 +3635,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data, if (err) goto out_bulk; + xe_bo_debug_mixed_mode_cur_index_advance(xe, bo); + args->handle = handle; goto out_put; diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h index 6340317f7d2e..c6d80e1bd6e7 100644 --- a/drivers/gpu/drm/xe/xe_bo.h +++ b/drivers/gpu/drm/xe/xe_bo.h @@ -52,6 +52,7 @@ #define XE_BO_FLAG_CPU_ADDR_MIRROR BIT(24) #define XE_BO_FLAG_FORCE_USER_VRAM BIT(25) #define XE_BO_FLAG_NO_COMPRESSION BIT(26) +#define XE_BO_FLAG_NEEDS_1G BIT(27) /* this one is trigger internally only */ #define XE_BO_FLAG_INTERNAL_TEST BIT(30) @@ -118,7 +119,8 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo, struct xe_tile *tile, struct dma_resv *resv, struct ttm_lru_bulk_move *bulk, size_t size, u16 cpu_caching, enum ttm_bo_type type, - u32 flags, struct drm_exec *exec); + u32 flags, struct dma_buf *dma_buf, + struct drm_exec *exec); struct xe_bo *xe_bo_create_locked(struct xe_device *xe, struct xe_tile *tile, struct xe_vm *vm, size_t size, enum ttm_bo_type type, u32 flags, diff --git a/drivers/gpu/drm/xe/xe_bo_types.h b/drivers/gpu/drm/xe/xe_bo_types.h index fcc63ae3f455..e45f24301050 100644 --- a/drivers/gpu/drm/xe/xe_bo_types.h +++ b/drivers/gpu/drm/xe/xe_bo_types.h @@ -36,6 +36,8 @@ struct xe_bo { struct xe_bo *backup_obj; /** @parent_obj: Ref to parent bo if this a backup_obj */ struct xe_bo *parent_obj; + /** @dma_buf: Imported dma-buf ref to keep its resv alive. */ + struct dma_buf *dma_buf; /** @flags: flags for this buffer object */ u32 flags; /** @vm: VM this BO is attached to, for extobj this will be NULL */ diff --git a/drivers/gpu/drm/xe/xe_configfs.c b/drivers/gpu/drm/xe/xe_configfs.c index 32102600a148..052cce962161 100644 --- a/drivers/gpu/drm/xe/xe_configfs.c +++ b/drivers/gpu/drm/xe/xe_configfs.c @@ -237,6 +237,18 @@ * * This setting only takes effect when probing the device. * + * Enable multi-queue + * ------------------ + * + * Multi-queue support on the device is enabled by default where the + * hardware supports it. Writing 0 force-disables multi-queue support: + * multi-queue exec-queue group creation via ioctl is refused, and the + * GuC feature is disabled:: + * + * # echo 0 > /sys/kernel/config/xe/0000:03:00.0/enable_multi_queue + * + * This attribute can only be set before binding to the device. + * * Remove devices * ============== * @@ -262,6 +274,7 @@ struct xe_config_group_device { struct wa_bb ctx_restore_mid_bb[XE_ENGINE_CLASS_MAX]; bool survivability_mode; bool enable_psmi; + bool enable_multi_queue; struct { unsigned int max_vfs; bool admin_only_pf; @@ -281,6 +294,7 @@ static const struct xe_config_device device_defaults = { .engines_allowed = U64_MAX, .survivability_mode = false, .enable_psmi = false, + .enable_multi_queue = true, .sriov = { .max_vfs = XE_DEFAULT_MAX_VFS, .admin_only_pf = XE_DEFAULT_ADMIN_ONLY_PF, @@ -575,6 +589,33 @@ static ssize_t enable_psmi_store(struct config_item *item, const char *page, siz return len; } +static ssize_t enable_multi_queue_show(struct config_item *item, char *page) +{ + struct xe_config_device *dev = to_xe_config_device(item); + + return sprintf(page, "%d\n", dev->enable_multi_queue); +} + +static ssize_t enable_multi_queue_store(struct config_item *item, const char *page, + size_t len) +{ + struct xe_config_group_device *dev = to_xe_config_group_device(item); + bool val; + int ret; + + ret = kstrtobool(page, &val); + if (ret) + return ret; + + guard(mutex)(&dev->lock); + if (is_bound(dev)) + return -EBUSY; + + dev->config.enable_multi_queue = val; + + return len; +} + static bool wa_bb_read_advance(bool dereference, char **p, const char *append, size_t len, size_t *max_size) @@ -812,6 +853,7 @@ static ssize_t ctx_restore_post_bb_store(struct config_item *item, CONFIGFS_ATTR(, ctx_restore_mid_bb); CONFIGFS_ATTR(, ctx_restore_post_bb); +CONFIGFS_ATTR(, enable_multi_queue); CONFIGFS_ATTR(, enable_psmi); CONFIGFS_ATTR(, engines_allowed); CONFIGFS_ATTR(, gt_types_allowed); @@ -820,6 +862,7 @@ CONFIGFS_ATTR(, survivability_mode); static struct configfs_attribute *xe_config_device_attrs[] = { &attr_ctx_restore_mid_bb, &attr_ctx_restore_post_bb, + &attr_enable_multi_queue, &attr_enable_psmi, &attr_engines_allowed, &attr_gt_types_allowed, @@ -1097,6 +1140,7 @@ static void dump_custom_dev_config(struct pci_dev *pdev, PRI_CUSTOM_ATTR("%llx", gt_types_allowed); PRI_CUSTOM_ATTR("%llx", engines_allowed); + PRI_CUSTOM_ATTR("%d", enable_multi_queue); PRI_CUSTOM_ATTR("%d", enable_psmi); PRI_CUSTOM_ATTR("%d", survivability_mode); PRI_CUSTOM_ATTR("%u", sriov.admin_only_pf); @@ -1226,6 +1270,27 @@ bool xe_configfs_get_psmi_enabled(struct pci_dev *pdev) } /** + * xe_configfs_get_enable_multi_queue - get configfs enable_multi_queue setting + * @pdev: pci device + * + * Return: true if multi-queue is enabled for this device (the default), + * false if it has been force-disabled via configfs. + */ +bool xe_configfs_get_enable_multi_queue(struct pci_dev *pdev) +{ + struct xe_config_group_device *dev = find_xe_config_group_device(pdev); + bool ret; + + if (!dev) + return true; + + ret = dev->config.enable_multi_queue; + config_group_put(&dev->group); + + return ret; +} + +/** * xe_configfs_get_ctx_restore_mid_bb - get configfs ctx_restore_mid_bb setting * @pdev: pci device * @class: hw engine class diff --git a/drivers/gpu/drm/xe/xe_configfs.h b/drivers/gpu/drm/xe/xe_configfs.h index 07d62bf0c152..4fbbeafba473 100644 --- a/drivers/gpu/drm/xe/xe_configfs.h +++ b/drivers/gpu/drm/xe/xe_configfs.h @@ -23,6 +23,7 @@ bool xe_configfs_primary_gt_allowed(struct pci_dev *pdev); bool xe_configfs_media_gt_allowed(struct pci_dev *pdev); u64 xe_configfs_get_engines_allowed(struct pci_dev *pdev); bool xe_configfs_get_psmi_enabled(struct pci_dev *pdev); +bool xe_configfs_get_enable_multi_queue(struct pci_dev *pdev); u32 xe_configfs_get_ctx_restore_mid_bb(struct pci_dev *pdev, enum xe_engine_class class, const u32 **cs); @@ -42,6 +43,7 @@ static inline bool xe_configfs_primary_gt_allowed(struct pci_dev *pdev) { return static inline bool xe_configfs_media_gt_allowed(struct pci_dev *pdev) { return true; } static inline u64 xe_configfs_get_engines_allowed(struct pci_dev *pdev) { return U64_MAX; } static inline bool xe_configfs_get_psmi_enabled(struct pci_dev *pdev) { return false; } +static inline bool xe_configfs_get_enable_multi_queue(struct pci_dev *pdev) { return true; } static inline u32 xe_configfs_get_ctx_restore_mid_bb(struct pci_dev *pdev, enum xe_engine_class class, const u32 **cs) { return 0; } diff --git a/drivers/gpu/drm/xe/xe_debugfs.c b/drivers/gpu/drm/xe/xe_debugfs.c index 3c018dbccc07..8de78cd0aa03 100644 --- a/drivers/gpu/drm/xe/xe_debugfs.c +++ b/drivers/gpu/drm/xe/xe_debugfs.c @@ -5,6 +5,7 @@ #include "xe_debugfs.h" +#include <linux/bits.h> #include <linux/debugfs.h> #include <linux/fault-inject.h> #include <linux/string_helpers.h> @@ -21,6 +22,7 @@ #include "xe_guc_ads.h" #include "xe_hw_engine.h" #include "xe_mmio.h" +#include "xe_pcode.h" #include "xe_pm.h" #include "xe_psmi.h" #include "xe_pxp_debugfs.h" @@ -41,6 +43,55 @@ DECLARE_FAULT_ATTR(gt_reset_failure); DECLARE_FAULT_ATTR(inject_csc_hw_error); +static bool csc_hw_error_available(struct xe_device *xe) +{ + return !IS_SRIOV_VF(xe) && xe->info.platform == XE_BATTLEMAGE; +} + +/* + * Fault injection table. Each entry registers a debugfs attribute; add a + * matching FAULT_ACTION() below for every entry added here. + */ +static struct { + const char *name; + struct fault_attr *attr; + bool (*is_visible)(struct xe_device *xe); +} xe_fault_inject_entry[] = { + { .name = "fail_gt_reset", + .attr = >_reset_failure }, + { .name = "inject_csc_hw_error", + .attr = &inject_csc_hw_error, + .is_visible = csc_hw_error_available }, +}; + +/* + * FAULT_ACTION(name, fault_attr) - generate xe_fault_<name>() accessor. + * Add one entry per row in xe_fault_inject_entry[]. + */ +#define FAULT_ACTION(name, fault_attr) \ +bool xe_fault_##name(void) \ +{ \ + return should_fail(&(fault_attr), 1); \ +} + +FAULT_ACTION(gt_reset, gt_reset_failure) +FAULT_ACTION(csc_hw_error, inject_csc_hw_error) + +static void xe_fault_inject_debugfs_register(struct xe_device *xe, + struct dentry *root) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(xe_fault_inject_entry); i++) { + if (xe_fault_inject_entry[i].is_visible && + !xe_fault_inject_entry[i].is_visible(xe)) + continue; + + fault_create_debugfs_attr(xe_fault_inject_entry[i].name, root, + xe_fault_inject_entry[i].attr); + } +} + static void read_residency_counter(struct xe_device *xe, struct xe_mmio *mmio, u32 offset, const char *name, struct drm_printer *p) { @@ -160,6 +211,22 @@ static int workaround_info(struct seq_file *m, void *data) return 0; } +static int pcode_info(struct seq_file *m, void *data) +{ + struct xe_device *xe = node_to_xe(m->private); + struct drm_printer p = drm_seq_file_printer(m); + struct xe_pcode_version version; + int ret = 0; + + ret = xe_get_pcode_version(xe, &version); + if (ret) + return ret; + + drm_printf(&p, "pcode version: %u.%u.%u\n", version.major, + version.minor, version.engg); + return 0; +} + static int dgfx_pkg_residencies_show(struct seq_file *m, void *data) { struct xe_device *xe; @@ -220,6 +287,10 @@ static const struct drm_info_list debugfs_list[] = { { .name = "workarounds", .show = workaround_info, }, }; +static const struct drm_info_list pcode_info_debugfs[] = { + { .name = "pcode_info", .show = pcode_info, }, +}; + static const struct drm_info_list debugfs_residencies[] = { { .name = "dgfx_pkg_residencies", .show = dgfx_pkg_residencies_show, }, { .name = "dgfx_pcie_link_residencies", .show = dgfx_pcie_link_residencies_show, }, @@ -543,6 +614,72 @@ static const struct file_operations disable_late_binding_fops = { .write = disable_late_binding_set, }; +#ifdef CONFIG_DRM_XE_DEBUG_PAGE_SIZE +static const char * const page_size_alloc_mode_names[] = { + [XE_PAGE_SIZE_ALLOC_CTRL_MODE_NONE] = "none", + [XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_2M] = "only_2m", + [XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_1G] = "only_1g", + [XE_PAGE_SIZE_ALLOC_CTRL_MODE_MIXED] = "mixed", +}; + +static ssize_t page_size_alloc_mode_show(struct file *f, char __user *ubuf, + size_t size, loff_t *pos) +{ + struct xe_device *xe = file_inode(f)->i_private; + char buf[32]; + int len; + enum xe_page_size_alloc_ctrl_mode mode; + + mode = READ_ONCE(xe->page_size_alloc_ctrl.mode); + if (mode >= ARRAY_SIZE(page_size_alloc_mode_names) || + !page_size_alloc_mode_names[mode]) + len = scnprintf(buf, sizeof(buf), "unknown\n"); + else + len = scnprintf(buf, sizeof(buf), "%s\n", + page_size_alloc_mode_names[mode]); + return simple_read_from_buffer(ubuf, size, pos, buf, len); +} + +static ssize_t page_size_alloc_mode_set(struct file *f, const char __user *ubuf, + size_t size, loff_t *pos) +{ + struct xe_device *xe = file_inode(f)->i_private; + int ret; + char buf[32]; + int mode; + + if (*pos) + return -ESPIPE; + + if (size > sizeof(buf) - 1) + return -EINVAL; + + ret = simple_write_to_buffer(buf, sizeof(buf) - 1, pos, ubuf, size); + if (ret < 0) + return ret; + buf[ret] = '\0'; + + mode = sysfs_match_string(page_size_alloc_mode_names, buf); + if (mode < 0) + return mode; + + mutex_lock(&xe->page_size_alloc_ctrl.lock); + if (mode == XE_PAGE_SIZE_ALLOC_CTRL_MODE_MIXED) + xe->page_size_alloc_ctrl.cur_index = 0; + WRITE_ONCE(xe->page_size_alloc_ctrl.mode, + (enum xe_page_size_alloc_ctrl_mode)mode); + mutex_unlock(&xe->page_size_alloc_ctrl.lock); + + return size; +} + +static const struct file_operations page_size_alloc_mode_fops = { + .owner = THIS_MODULE, + .read = page_size_alloc_mode_show, + .write = page_size_alloc_mode_set, +}; +#endif + void xe_debugfs_register(struct xe_device *xe) { struct ttm_device *bdev = &xe->ttm; @@ -562,10 +699,20 @@ void xe_debugfs_register(struct xe_device *xe) drm_debugfs_create_files(debugfs_residencies, ARRAY_SIZE(debugfs_residencies), root, minor); - fault_create_debugfs_attr("inject_csc_hw_error", root, - &inject_csc_hw_error); } + /* + * Pcode version read from PMT is currently only supported on CRI and BMG platforms in PF + * mode, as both platforms support the necessary telemetry read mechanism and have a fixed + * PUNIT_VERSION_OFFSET. + * Attempting this access on other platforms must be verified before enabling support. + */ + if (!IS_SRIOV_VF(xe) && + (xe->info.platform == XE_CRESCENTISLAND || xe->info.platform == XE_BATTLEMAGE)) + drm_debugfs_create_files(pcode_info_debugfs, + ARRAY_SIZE(pcode_info_debugfs), + root, minor); + debugfs_create_file("forcewake_all", 0400, root, xe, &forcewake_all_fops); @@ -584,6 +731,18 @@ void xe_debugfs_register(struct xe_device *xe) debugfs_create_file("disable_late_binding", 0600, root, xe, &disable_late_binding_fops); +#ifdef CONFIG_DRM_XE_DEBUG_PAGE_SIZE + /* + * Expose a debugfs knob to control user BO page-size allocation: + * "none" - default behavior + * "only_2m" - force 2M page allocations + * "only_1g" - force 1G page allocations + * "mixed" - select 4K, 64K, 2M, and 1G in round-robin order + */ + if (xe_debug_page_size_supported(xe)) + debugfs_create_file("page_size_alloc_mode", 0600, root, xe, + &page_size_alloc_mode_fops); +#endif /* * Don't expose page reclaim configuration file if not supported by the * hardware initially. @@ -609,7 +768,7 @@ void xe_debugfs_register(struct xe_device *xe) xe_psmi_debugfs_register(xe); - fault_create_debugfs_attr("fail_gt_reset", root, >_reset_failure); + xe_fault_inject_debugfs_register(xe, root); if (IS_SRIOV_PF(xe)) xe_sriov_pf_debugfs_register(xe, root); diff --git a/drivers/gpu/drm/xe/xe_debugfs.h b/drivers/gpu/drm/xe/xe_debugfs.h index 17f4c2f1b5e4..cd56f7442b99 100644 --- a/drivers/gpu/drm/xe/xe_debugfs.h +++ b/drivers/gpu/drm/xe/xe_debugfs.h @@ -6,11 +6,17 @@ #ifndef _XE_DEBUGFS_H_ #define _XE_DEBUGFS_H_ +#include <linux/types.h> + struct xe_device; #ifdef CONFIG_DEBUG_FS +bool xe_fault_gt_reset(void); +bool xe_fault_csc_hw_error(void); void xe_debugfs_register(struct xe_device *xe); #else +static inline bool xe_fault_gt_reset(void) { return false; } +static inline bool xe_fault_csc_hw_error(void) { return false; } static inline void xe_debugfs_register(struct xe_device *xe) { } #endif diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c index ad7f3e61d457..7007b6113760 100644 --- a/drivers/gpu/drm/xe/xe_device.c +++ b/drivers/gpu/drm/xe/xe_device.c @@ -427,7 +427,6 @@ static const struct drm_ioctl_desc xe_ioctls_admin_only[] = { static const struct drm_driver admin_only_driver = { .driver_features = - XE_DISPLAY_DRIVER_FEATURES | DRIVER_GEM | DRIVER_RENDER, .open = xe_file_open, .postclose = xe_file_close, @@ -439,7 +438,6 @@ static const struct drm_driver admin_only_driver = { .major = DRIVER_MAJOR, .minor = DRIVER_MINOR, .patchlevel = DRIVER_PATCHLEVEL, - XE_DISPLAY_DRIVER_OPS, }; /** @@ -581,7 +579,7 @@ int xe_device_init_early(struct xe_device *xe) WQ_MEM_RECLAIM); xe->ordered_wq = alloc_ordered_workqueue("xe-ordered-wq", 0); xe->unordered_wq = alloc_workqueue("xe-unordered-wq", WQ_PERCPU, 0); - xe->destroy_wq = alloc_workqueue("xe-destroy-wq", WQ_PERCPU, 0); + xe->destroy_wq = alloc_workqueue("xe-destroy-wq", WQ_PERCPU | WQ_MEM_RECLAIM, 0); if (!xe->ordered_wq || !xe->unordered_wq || !xe->preempt_fence_wq || !xe->destroy_wq) { /* @@ -923,6 +921,27 @@ static void xe_device_wedged_fini(struct drm_device *drm, void *arg) xe_pm_runtime_put(xe); } +#ifdef CONFIG_DRM_XE_DEBUG_PAGE_SIZE +static int xe_debug_page_size_alloc_ctrl_init(struct xe_device *xe) +{ + int err; + + err = drmm_mutex_init(&xe->drm, &xe->page_size_alloc_ctrl.lock); + if (err) + return err; + + xe->page_size_alloc_ctrl.mode = XE_PAGE_SIZE_ALLOC_CTRL_MODE_NONE; + xe->page_size_alloc_ctrl.cur_index = 0; + + return 0; +} +#else +static int xe_debug_page_size_alloc_ctrl_init(struct xe_device *xe) +{ + return 0; +} +#endif + int xe_device_probe(struct xe_device *xe) { struct xe_tile *tile; @@ -1075,6 +1094,10 @@ int xe_device_probe(struct xe_device *xe) if (err) return err; + err = xe_debug_page_size_alloc_ctrl_init(xe); + if (err) + return err; + err = drm_dev_register(&xe->drm, 0); if (err) return err; diff --git a/drivers/gpu/drm/xe/xe_device.h b/drivers/gpu/drm/xe/xe_device.h index a03760d0ce38..6c4cfaebc44a 100644 --- a/drivers/gpu/drm/xe/xe_device.h +++ b/drivers/gpu/drm/xe/xe_device.h @@ -212,6 +212,54 @@ static inline bool xe_device_wedged(struct xe_device *xe) return atomic_read(&xe->wedged.flag); } +#ifdef CONFIG_DRM_XE_DEBUG_PAGE_SIZE +static inline bool xe_debug_page_size_supported(struct xe_device *xe) +{ + return IS_DGFX(xe); +} + +static inline bool xe_debug_page_size_mode_not_none(struct xe_device *xe) +{ + enum xe_page_size_alloc_ctrl_mode mode; + + if (!xe_debug_page_size_supported(xe)) + return false; + + mode = READ_ONCE(xe->page_size_alloc_ctrl.mode); + + return mode == XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_2M || + mode == XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_1G || + mode == XE_PAGE_SIZE_ALLOC_CTRL_MODE_MIXED; +} + +static inline bool xe_debug_page_size_mode_is_mixed(struct xe_device *xe) +{ + enum xe_page_size_alloc_ctrl_mode mode; + + if (!xe_debug_page_size_supported(xe)) + return false; + + mode = READ_ONCE(xe->page_size_alloc_ctrl.mode); + + return mode == XE_PAGE_SIZE_ALLOC_CTRL_MODE_MIXED; +} +#else +static inline bool xe_debug_page_size_supported(struct xe_device *xe) +{ + return false; +} + +static inline bool xe_debug_page_size_mode_not_none(struct xe_device *xe) +{ + return false; +} + +static inline bool xe_debug_page_size_mode_is_mixed(struct xe_device *xe) +{ + return false; +} +#endif + void xe_device_set_wedged_method(struct xe_device *xe, unsigned long method); void xe_device_declare_wedged(struct xe_device *xe); int xe_device_validate_wedged_mode(struct xe_device *xe, unsigned int mode); diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h index 022e08205897..03a7bb08adf7 100644 --- a/drivers/gpu/drm/xe/xe_device_types.h +++ b/drivers/gpu/drm/xe/xe_device_types.h @@ -61,6 +61,23 @@ enum xe_wedged_mode { XE_WEDGED_MODE_UPON_ANY_HANG_NO_RESET = 2, }; +#ifdef CONFIG_DRM_XE_DEBUG_PAGE_SIZE +/** + * enum xe_page_size_alloc_ctrl_mode - User BO page-size allocation control modes + * @XE_PAGE_SIZE_ALLOC_CTRL_MODE_NONE: Use the normal allocation policy + * @XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_2M: Force user BO allocations to 2M pages + * @XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_1G: Force user BO allocations to 1G pages + * @XE_PAGE_SIZE_ALLOC_CTRL_MODE_MIXED: Select page sizes in round-robin order + * (4K, 64K, 2M, 1G) + */ +enum xe_page_size_alloc_ctrl_mode { + XE_PAGE_SIZE_ALLOC_CTRL_MODE_NONE = 0, + XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_2M, + XE_PAGE_SIZE_ALLOC_CTRL_MODE_ONLY_1G, + XE_PAGE_SIZE_ALLOC_CTRL_MODE_MIXED +}; +#endif + #define XE_BO_INVALID_OFFSET LONG_MAX #define GRAPHICS_VER(xe) ((xe)->info.graphics_verx100 / 100) @@ -355,7 +372,7 @@ struct xe_device { /** @unordered_wq: used to serialize unordered work */ struct workqueue_struct *unordered_wq; - /** @destroy_wq: used to serialize user destroy work, like queue */ + /** @destroy_wq: used to serialize SVM pagemap destroy work */ struct workqueue_struct *destroy_wq; /** @tiles: device tiles */ @@ -404,6 +421,10 @@ struct xe_device { const struct xe_pat_table_entry *pat_primary_pta; /** @pat.pat_media_pta: media GT PAT entry for page table accesses */ const struct xe_pat_table_entry *pat_media_pta; + /** @pat.pat_primary_tr_pta: primary GT PAT entry for TRTT page table accesses */ + const struct xe_pat_table_entry *pat_primary_tr_pta; + /** @pat.pat_media_tr_pta: media GT PAT entry for TRTT page table accesses */ + const struct xe_pat_table_entry *pat_media_tr_pta; u16 idx[__XE_CACHE_LEVEL_COUNT]; } pat; @@ -474,6 +495,20 @@ struct xe_device { /** @late_bind: xe mei late bind interface */ struct xe_late_bind late_bind; +#ifdef CONFIG_DRM_XE_DEBUG_PAGE_SIZE + /** + * @page_size_alloc_ctrl: User BO page-size allocation + * debug control state + */ + struct { + /** @page_size_alloc_ctrl.mode: xe page size allocation control mode */ + enum xe_page_size_alloc_ctrl_mode mode; + /** @page_size_alloc_ctrl.cur_index: Round-robin index used by mixed mode */ + u32 cur_index; + /** @page_size_alloc_ctrl.lock: Protects @mode and @cur_index */ + struct mutex lock; + } page_size_alloc_ctrl; +#endif /** @oa: oa observation subsystem */ struct xe_oa oa; diff --git a/drivers/gpu/drm/xe/xe_dma_buf.c b/drivers/gpu/drm/xe/xe_dma_buf.c index 8a920e58245c..bf0728838ead 100644 --- a/drivers/gpu/drm/xe/xe_dma_buf.c +++ b/drivers/gpu/drm/xe/xe_dma_buf.c @@ -302,7 +302,7 @@ xe_dma_buf_create_obj(struct drm_device *dev, struct dma_buf *dma_buf) bo = xe_bo_init_locked(xe, NULL, NULL, resv, NULL, dma_buf->size, 0, /* Will require 1way or 2way for vm_bind */ - ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, &exec); + ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, dma_buf, &exec); drm_exec_retry_on_contention(&exec); if (IS_ERR(bo)) { ret = PTR_ERR(bo); diff --git a/drivers/gpu/drm/xe/xe_eu_stall.c b/drivers/gpu/drm/xe/xe_eu_stall.c index d37770c58c5d..8a7c1b5d5ab9 100644 --- a/drivers/gpu/drm/xe/xe_eu_stall.c +++ b/drivers/gpu/drm/xe/xe_eu_stall.c @@ -7,6 +7,7 @@ #include <linux/fs.h> #include <linux/poll.h> #include <linux/types.h> +#include <linux/iopoll.h> #include <drm/drm_drv.h> #include <generated/xe_wa_oob.h> @@ -20,6 +21,7 @@ #include "xe_gt_printk.h" #include "xe_gt_topology.h" #include "xe_macros.h" +#include "xe_mmio.h" #include "xe_observation.h" #include "xe_pm.h" #include "xe_trace.h" @@ -27,8 +29,16 @@ #include "regs/xe_eu_stall_regs.h" #include "regs/xe_gt_regs.h" +#include "regs/xe_regs.h" #define POLL_PERIOD_MS 5 +#define FW_WA_WAIT_TIMEOUT_US 10000 + +#define SWF_EUSTALL_MASK REG_GENMASK(6, 5) +#define REQ_EUSTALL_ENABLE REG_BIT(5) +#define ACK_EUSTALL_ENABLE REG_GENMASK(6, 5) +#define REQ_EUSTALL_DISABLE REG_BIT(6) +#define ACK_EUSTALL_DISABLE 0 static size_t per_xecore_buf_size = SZ_512K; @@ -682,7 +692,7 @@ static int xe_eu_stall_stream_enable(struct xe_eu_stall_data_stream *stream) struct per_xecore_buf *xecore_buf; struct xe_gt *gt = stream->gt; u16 group, instance; - int xecore; + int xecore, ret = 0; /* Take runtime pm ref and forcewake to disable RC6 */ xe_pm_runtime_get(gt_to_xe(gt)); @@ -693,6 +703,18 @@ static int xe_eu_stall_stream_enable(struct xe_eu_stall_data_stream *stream) return -ETIMEDOUT; } + if (XE_GT_WA(gt, 14027054324)) { + /* Request the firmware to apply the workaround and wait for an ACK */ + xe_mmio_write32(>->mmio, SWF_SCRATCHPAD(0), REQ_EUSTALL_ENABLE); + ret = xe_mmio_wait32(>->mmio, SWF_SCRATCHPAD(0), SWF_EUSTALL_MASK, + ACK_EUSTALL_ENABLE, FW_WA_WAIT_TIMEOUT_US, NULL, false); + if (ret) { + xe_gt_err(gt, "Timeout polling for EU stall enable ACK from firmware\n"); + xe_force_wake_put(gt_to_fw(gt), stream->fw_ref); + xe_pm_runtime_put(gt_to_xe(gt)); + return ret; + } + } if (XE_GT_WA(gt, 22016596838)) xe_gt_mcr_multicast_write(gt, ROW_CHICKEN2, REG_MASKED_FIELD_ENABLE(DISABLE_DOP_GATING)); @@ -730,7 +752,7 @@ static int xe_eu_stall_stream_enable(struct xe_eu_stall_data_stream *stream) reg_value |= XEHPC_EUSTALL_BASE_ENABLE_SAMPLING; xe_gt_mcr_multicast_write(gt, XEHPC_EUSTALL_BASE, reg_value); - return 0; + return ret; } static void eu_stall_data_buf_poll_work_fn(struct work_struct *work) @@ -840,6 +862,7 @@ static int xe_eu_stall_enable_locked(struct xe_eu_stall_data_stream *stream) static int xe_eu_stall_disable_locked(struct xe_eu_stall_data_stream *stream) { struct xe_gt *gt = stream->gt; + int ret = 0; if (!stream->enabled) return 0; @@ -853,11 +876,19 @@ static int xe_eu_stall_disable_locked(struct xe_eu_stall_data_stream *stream) if (XE_GT_WA(gt, 22016596838)) xe_gt_mcr_multicast_write(gt, ROW_CHICKEN2, REG_MASKED_FIELD_DISABLE(DISABLE_DOP_GATING)); + if (XE_GT_WA(gt, 14027054324)) { + /* Request the firmware to revert the workaround and wait for an ACK */ + xe_mmio_write32(>->mmio, SWF_SCRATCHPAD(0), REQ_EUSTALL_DISABLE); + ret = xe_mmio_wait32(>->mmio, SWF_SCRATCHPAD(0), SWF_EUSTALL_MASK, + ACK_EUSTALL_DISABLE, FW_WA_WAIT_TIMEOUT_US, NULL, false); + if (ret) + xe_gt_err(gt, "Timeout polling for EU stall disable ACK from firmware\n"); + } xe_force_wake_put(gt_to_fw(gt), stream->fw_ref); xe_pm_runtime_put(gt_to_xe(gt)); - return 0; + return ret; } static long xe_eu_stall_stream_ioctl_locked(struct xe_eu_stall_data_stream *stream, diff --git a/drivers/gpu/drm/xe/xe_exec.c b/drivers/gpu/drm/xe/xe_exec.c index e05dabfcd43c..d5293bc33a67 100644 --- a/drivers/gpu/drm/xe/xe_exec.c +++ b/drivers/gpu/drm/xe/xe_exec.c @@ -292,13 +292,23 @@ retry: goto err_exec; } - /* Wait behind rebinds */ + /* + * Wait behind rebinds and any kernel operations (evictions, defrag + * moves, ...) on the VM and all external BOs. The VM's private BOs + * carry their kernel ops in the VM dma-resv KERNEL slot, while each + * external BO carries them in its own dma-resv KERNEL slot; both are + * covered by iterating every object locked by the exec, mirroring the + * drm_gpuvm_resv_add_fence() below. + */ if (!xe_vm_in_lr_mode(vm)) { - err = xe_sched_job_add_deps(job, - xe_vm_resv(vm), - DMA_RESV_USAGE_KERNEL); - if (err) - goto err_put_job; + struct drm_gem_object *obj; + + drm_exec_for_each_locked_object(exec, obj) { + err = xe_sched_job_add_deps(job, obj->resv, + DMA_RESV_USAGE_KERNEL); + if (err) + goto err_put_job; + } } for (i = 0; i < num_syncs && !err; i++) diff --git a/drivers/gpu/drm/xe/xe_exec_queue.c b/drivers/gpu/drm/xe/xe_exec_queue.c index 1b5ca3ce578a..38972b6e6d37 100644 --- a/drivers/gpu/drm/xe/xe_exec_queue.c +++ b/drivers/gpu/drm/xe/xe_exec_queue.c @@ -530,10 +530,7 @@ struct xe_exec_queue *xe_exec_queue_create_bind(struct xe_device *xe, migrate_vm = xe_migrate_get_vm(tile->migrate); if (xe->info.has_usm) { - struct xe_hw_engine *hwe = xe_gt_hw_engine(gt, - XE_ENGINE_CLASS_COPY, - gt->usm.reserved_bcs_instance, - false); + struct xe_hw_engine *hwe = gt->usm.paging_hwe0; if (!hwe) { xe_vm_put(migrate_vm); @@ -842,6 +839,7 @@ static int xe_exec_queue_group_init(struct xe_device *xe, struct xe_exec_queue * group->primary = q; group->cgp_bo = bo; INIT_LIST_HEAD(&group->list); + spin_lock_init(&group->suspend_lock); xa_init_flags(&group->xa, XA_FLAGS_ALLOC1); mutex_init(&group->list_lock); q->multi_queue.group = group; @@ -1056,6 +1054,7 @@ int xe_exec_queue_set_property_ioctl(struct drm_device *dev, void *data, static int exec_queue_user_ext_check(struct xe_exec_queue *q, u64 properties) { + struct xe_device *xe = gt_to_xe(q->gt); u64 secondary_queue_valid_props = BIT_ULL(DRM_XE_EXEC_QUEUE_SET_PROPERTY_MULTI_GROUP) | BIT_ULL(DRM_XE_EXEC_QUEUE_SET_PROPERTY_MULTI_QUEUE_PRIORITY); @@ -1067,6 +1066,16 @@ static int exec_queue_user_ext_check(struct xe_exec_queue *q, u64 properties) properties & ~secondary_queue_valid_props) return -EINVAL; + /* + * HWDRM is the only supported PXP type today. It is display related and + * hence can't work with multi-queue. Reject the combination. The secondary + * queue path above already rejects any PXP property, so this also covers + * the multi-queue primary which would otherwise allow it. + */ + if (XE_IOCTL_DBG(xe, (properties & BIT_ULL(DRM_XE_EXEC_QUEUE_SET_PROPERTY_MULTI_GROUP)) && + (properties & BIT_ULL(DRM_XE_EXEC_QUEUE_SET_PROPERTY_PXP_TYPE)))) + return -EINVAL; + return 0; } diff --git a/drivers/gpu/drm/xe/xe_exec_queue_types.h b/drivers/gpu/drm/xe/xe_exec_queue_types.h index d27ce24daae5..53b6c0bf4849 100644 --- a/drivers/gpu/drm/xe/xe_exec_queue_types.h +++ b/drivers/gpu/drm/xe/xe_exec_queue_types.h @@ -62,6 +62,12 @@ struct xe_exec_queue_group { struct list_head list; /** @list_lock: Secondary queue list lock */ struct mutex list_lock; + /** + * @suspend_lock: Makes a secondary's suspend/resume and its forwarding + * to the primary atomic. Nested outside of the queue's message lock + * (@xe_guc_exec_queue.sched.msg_lock). + */ + spinlock_t suspend_lock; /** @sync_pending: CGP_SYNC_DONE g2h response pending */ bool sync_pending; /** @banned: Group banned */ @@ -200,6 +206,18 @@ struct xe_exec_queue { u32 seqno; /** @lr.link: link into VM's list of exec queues */ struct list_head link; + /** + * @lr.suspended: Tracks whether the consumer-issued suspend() + * succeeded and a matching resume() is still owed. suspend() can + * fail (e.g. killed/banned/wedged), leaving the queue + * un-suspended, so consumers must only resume() queues that were + * actually suspended. Set by the suspend caller on success and + * cleared by the resume caller. A queue is only ever suspended by + * a single consumer at a time (preempt-fence mode and hw engine + * group fault mode are mutually exclusive), so a single flag is + * sufficient. + */ + bool suspended; } lr; #define XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT 0 @@ -311,6 +329,15 @@ struct xe_exec_queue_ops { */ int (*suspend_wait)(struct xe_exec_queue *q); /** + * @suspend_wait_blocking: Like @suspend_wait, but waits uninterruptibly + * (does not abort on the calling task's signals). For cleanup/undo paths + * that must complete a suspend on behalf of a queue that may belong to a + * different process than the caller: a signal to the caller must not + * abandon the wait, which would leave the other process's queue + * suspended forever (cross-process DoS). A timeout bans like suspend_wait. + */ + int (*suspend_wait_blocking)(struct xe_exec_queue *q); + /** * @resume: Resume exec queue execution, exec queue must be in a suspended * state and dma fence returned from most recent suspend call must be * signalled when this function is called. diff --git a/drivers/gpu/drm/xe/xe_execlist.c b/drivers/gpu/drm/xe/xe_execlist.c index 6b86b4f9cc1c..cc33ae80e8cf 100644 --- a/drivers/gpu/drm/xe/xe_execlist.c +++ b/drivers/gpu/drm/xe/xe_execlist.c @@ -468,6 +468,7 @@ static const struct xe_exec_queue_ops execlist_exec_queue_ops = { .set_preempt_timeout = execlist_exec_queue_set_preempt_timeout, .suspend = execlist_exec_queue_suspend, .suspend_wait = execlist_exec_queue_suspend_wait, + .suspend_wait_blocking = execlist_exec_queue_suspend_wait, .resume = execlist_exec_queue_resume, .reset_status = execlist_exec_queue_reset_status, }; diff --git a/drivers/gpu/drm/xe/xe_gt.c b/drivers/gpu/drm/xe/xe_gt.c index d904527a8898..dfdacc0f6de9 100644 --- a/drivers/gpu/drm/xe/xe_gt.c +++ b/drivers/gpu/drm/xe/xe_gt.c @@ -21,6 +21,7 @@ #include "regs/xe_gt_regs.h" #include "xe_assert.h" #include "xe_bb.h" +#include "xe_debugfs.h" #include "xe_device.h" #include "xe_eu_stall.h" #include "xe_exec_queue.h" @@ -926,7 +927,7 @@ static void gt_reset_worker(struct work_struct *w) xe_gt_info(gt, "reset started\n"); - if (xe_fault_inject_gt_reset()) { + if (xe_fault_gt_reset()) { err = -ECANCELED; goto err_fail; } @@ -986,7 +987,7 @@ void xe_gt_reset_async(struct xe_gt *gt) return; /* Don't do a reset while one is already in flight */ - if (!xe_fault_inject_gt_reset() && xe_uc_reset_prepare(>->uc)) + if (!xe_fault_gt_reset() && xe_uc_reset_prepare(>->uc)) return; xe_gt_info(gt, "reset queued from %ps\n", __builtin_return_address(0)); diff --git a/drivers/gpu/drm/xe/xe_gt.h b/drivers/gpu/drm/xe/xe_gt.h index 4150aa594f05..65a4655b0994 100644 --- a/drivers/gpu/drm/xe/xe_gt.h +++ b/drivers/gpu/drm/xe/xe_gt.h @@ -6,8 +6,6 @@ #ifndef _XE_GT_H_ #define _XE_GT_H_ -#include <linux/fault-inject.h> - #include <drm/drm_util.h> #include "xe_device.h" @@ -38,12 +36,6 @@ xe_gt_is_media_type(gt_) ? MEDIA_VER(xe) : GRAPHICS_VER(xe); \ }) -extern struct fault_attr gt_reset_failure; -static inline bool xe_fault_inject_gt_reset(void) -{ - return IS_ENABLED(CONFIG_DEBUG_FS) && should_fail(>_reset_failure, 1); -} - struct xe_gt *xe_gt_alloc(struct xe_tile *tile); int xe_gt_init_early(struct xe_gt *gt); int xe_gt_init(struct xe_gt *gt); @@ -137,10 +129,8 @@ static inline bool xe_gt_is_media_type(struct xe_gt *gt) static inline bool xe_gt_is_usm_hwe(struct xe_gt *gt, struct xe_hw_engine *hwe) { - struct xe_device *xe = gt_to_xe(gt); - - return xe->info.has_usm && hwe->class == XE_ENGINE_CLASS_COPY && - hwe->instance == gt->usm.reserved_bcs_instance; + return hwe->class == XE_ENGINE_CLASS_COPY && + (gt->usm.paging_logical_mask & BIT(hwe->logical_instance)); } /** diff --git a/drivers/gpu/drm/xe/xe_gt_sriov_pf_debugfs.c b/drivers/gpu/drm/xe/xe_gt_sriov_pf_debugfs.c index ffa27f66bba7..0f242db775e1 100644 --- a/drivers/gpu/drm/xe/xe_gt_sriov_pf_debugfs.c +++ b/drivers/gpu/drm/xe/xe_gt_sriov_pf_debugfs.c @@ -381,10 +381,11 @@ static ssize_t sched_group_engines_read(struct file *file, char __user *buf, if (group < num_groups) { for_each_hw_engine(hwe, gt, id) { - u8 guc_class = xe_engine_class_to_guc_class(hwe->class); + u8 guc_class = xe_hwe_to_guc_class(hwe); + u16 guc_logical_instance = xe_hwe_guc_logical_instance(hwe); u32 mask = groups[group].engines[guc_class]; - if (mask & BIT(hwe->logical_instance)) { + if (mask & BIT(guc_logical_instance)) { strlcat(engines, hwe->name, sizeof(engines)); strlcat(engines, " ", sizeof(engines)); } diff --git a/drivers/gpu/drm/xe/xe_gt_sriov_pf_policy.c b/drivers/gpu/drm/xe/xe_gt_sriov_pf_policy.c index e8458d63742d..cdfe194926d3 100644 --- a/drivers/gpu/drm/xe/xe_gt_sriov_pf_policy.c +++ b/drivers/gpu/drm/xe/xe_gt_sriov_pf_policy.c @@ -471,7 +471,8 @@ static void pf_sched_group_media_slices(struct xe_gt *gt, struct guc_sched_group return; for_each_hw_engine(hwe, gt, id) { - u8 guc_class = xe_engine_class_to_guc_class(hwe->class); + u8 guc_class = xe_hwe_to_guc_class(hwe); + u16 guc_logical_instance = xe_hwe_guc_logical_instance(hwe); switch (hwe->class) { case XE_ENGINE_CLASS_VIDEO_DECODE: @@ -490,7 +491,7 @@ static void pf_sched_group_media_slices(struct xe_gt *gt, struct guc_sched_group slice = 0; } - values[slice_to_group[slice]].engines[guc_class] |= BIT(hwe->logical_instance); + values[slice_to_group[slice]].engines[guc_class] |= BIT(guc_logical_instance); } *groups = values; diff --git a/drivers/gpu/drm/xe/xe_gt_sriov_vf.c b/drivers/gpu/drm/xe/xe_gt_sriov_vf.c index 0cd9d77f3351..37899fcf5b22 100644 --- a/drivers/gpu/drm/xe/xe_gt_sriov_vf.c +++ b/drivers/gpu/drm/xe/xe_gt_sriov_vf.c @@ -658,6 +658,33 @@ static int vf_cache_sched_groups_status(struct xe_gt *gt) return 0; } +static int vf_cache_num_paging_engines(struct xe_gt *gt) +{ + struct xe_guc *guc = >->uc.guc; + struct xe_uc_fw_version guc_version; + u32 value = 0; + int err; + + xe_gt_sriov_vf_guc_versions(gt, NULL, &guc_version); + + if (MAKE_GUC_VER_STRUCT(guc_version) < MAKE_GUC_VER(1, 36, 0)) + return 0; + + err = guc_action_query_single_klv32(guc, GUC_KLV_GLOBAL_CFG_NUM_PAGING_ENGINE_INSTANCES_KEY, + &value); + if (unlikely(err)) { + xe_gt_sriov_err(gt, + "Failed to obtain the number of paging instances (%pe)\n", + ERR_PTR(err)); + return err; + } + + gt->sriov.vf.runtime.num_paging_engine_instances = value; + + xe_gt_sriov_dbg(gt, "num_paging_engines %u\n", value); + return 0; +} + /** * xe_gt_sriov_vf_query_config - Query SR-IOV config data over MMIO. * @gt: the &xe_gt @@ -694,6 +721,10 @@ int xe_gt_sriov_vf_query_config(struct xe_gt *gt) if (has_gmdid(xe)) vf_cache_gmdid(gt); + err = vf_cache_num_paging_engines(gt); + if (unlikely(err)) + return err; + return 0; } @@ -731,6 +762,22 @@ u16 xe_gt_sriov_vf_guc_ids(struct xe_gt *gt) return gt->sriov.vf.self_config.num_ctxs; } +/** + * xe_gt_sriov_vf_paging_engines - Return the number of paging engine instances + * @gt: the &xe_gt + * + * This function is for VF use only. + * + * Return: number of GuC paging engine instances configured by the PF. + */ +u32 xe_gt_sriov_vf_paging_engines(struct xe_gt *gt) +{ + xe_gt_assert(gt, IS_SRIOV_VF(gt_to_xe(gt))); + xe_gt_assert(gt, gt->sriov.vf.guc_version.major); + + return gt->sriov.vf.runtime.num_paging_engine_instances; +} + static int relay_action_handshake(struct xe_gt *gt, u32 *major, u32 *minor) { u32 request[VF2PF_HANDSHAKE_REQUEST_MSG_LEN] = { diff --git a/drivers/gpu/drm/xe/xe_gt_sriov_vf.h b/drivers/gpu/drm/xe/xe_gt_sriov_vf.h index 79878f21b1da..d171a8242a34 100644 --- a/drivers/gpu/drm/xe/xe_gt_sriov_vf.h +++ b/drivers/gpu/drm/xe/xe_gt_sriov_vf.h @@ -31,6 +31,7 @@ u32 xe_gt_sriov_vf_gmdid(struct xe_gt *gt); u16 xe_gt_sriov_vf_guc_ids(struct xe_gt *gt); u64 xe_gt_sriov_vf_lmem(struct xe_gt *gt); bool xe_gt_sriov_vf_sched_groups_enabled(struct xe_gt *gt); +u32 xe_gt_sriov_vf_paging_engines(struct xe_gt *gt); u32 xe_gt_sriov_vf_read32(struct xe_gt *gt, struct xe_reg reg); void xe_gt_sriov_vf_write32(struct xe_gt *gt, struct xe_reg reg, u32 val); diff --git a/drivers/gpu/drm/xe/xe_gt_sriov_vf_types.h b/drivers/gpu/drm/xe/xe_gt_sriov_vf_types.h index 80562ffadb16..466f0abd9c28 100644 --- a/drivers/gpu/drm/xe/xe_gt_sriov_vf_types.h +++ b/drivers/gpu/drm/xe/xe_gt_sriov_vf_types.h @@ -29,6 +29,10 @@ struct xe_gt_sriov_vf_runtime { u32 gmdid; /** @uses_sched_groups: whether PF enabled sched groups or not. */ bool uses_sched_groups; + /** + * @num_paging_engine_instances: number of configured paging engines. + */ + u32 num_paging_engine_instances; /** @regs_size: size of runtime register array. */ u32 regs_size; /** @num_regs: number of runtime registers in the array. */ diff --git a/drivers/gpu/drm/xe/xe_gt_types.h b/drivers/gpu/drm/xe/xe_gt_types.h index e5588c88800a..a8bbfbdf3849 100644 --- a/drivers/gpu/drm/xe/xe_gt_types.h +++ b/drivers/gpu/drm/xe/xe_gt_types.h @@ -145,6 +145,11 @@ struct xe_gt { /** @info.has_indirect_ring_state: GT has indirect ring state support */ u8 has_indirect_ring_state:1; /** + * @info.has_uncorrectable_error_reporting: GT has uncorrectable + * error reporting support + */ + u8 has_uncorrectable_error_reporting:1; + /** * @info.has_xe2_blt_instructions: GT supports Xe2-style MEM_SET * and MEM_COPY blitter functionality. Note that despite the * name, some Xe1 platforms may also support this "Xe2-style" @@ -230,10 +235,16 @@ struct xe_gt { */ struct xe_sa_manager *bb_pool; /** - * @usm.reserved_bcs_instance: reserved BCS instance used for USM - * operations (e.g. migrations, fixing page tables) + * @usm.paging_hwe0: The first designated paging engine. + * This is some reserved BCS instance used for USM operations + * (e.g. migrations, fixing page tables) + */ + struct xe_hw_engine *paging_hwe0; + /** + * @usm.paging_logical_mask: logical mask of paging engines. + * Should be densely populated. */ - u16 reserved_bcs_instance; + u32 paging_logical_mask; } usm; /** @ordered_wq: used to serialize GT resets and TDRs */ diff --git a/drivers/gpu/drm/xe/xe_guc.c b/drivers/gpu/drm/xe/xe_guc.c index 4023700ff2a9..4286bd05c686 100644 --- a/drivers/gpu/drm/xe/xe_guc.c +++ b/drivers/gpu/drm/xe/xe_guc.c @@ -12,6 +12,7 @@ #include "abi/guc_actions_abi.h" #include "abi/guc_errors_abi.h" +#include "abi/guc_klvs_abi.h" #include "regs/xe_gt_regs.h" #include "regs/xe_gtt_defs.h" #include "regs/xe_guc_regs.h" @@ -101,6 +102,14 @@ static u32 guc_ctl_feature_flags(struct xe_guc *guc) if (xe_device_is_l2_flush_optimized(xe) && xe_gt_is_media_type(guc_to_gt(guc))) flags |= GUC_CTL_ENABLE_L2FLUSH_OPT; + /* + * On GuC firmware 70.66 and above, the GUC_FEATURE_KLV_DISABLE_MULTI_QUEUE + * Feature KLV is used instead. + */ + if (!xe_configfs_get_enable_multi_queue(to_pci_dev(xe->drm.dev)) && + !GUC_FIRMWARE_VER_AT_LEAST(guc, 70, 66)) + flags |= GUC_CTL_DISABLE_MULTI_QUEUE; + return flags; } @@ -641,6 +650,15 @@ int xe_guc_opt_in_features_enable(struct xe_guc *guc) if (GUC_SUBMIT_VER(guc) >= MAKE_GUC_VER(1, 7, 0)) klvs[count++] = PREP_GUC_KLV_TAG(OPT_IN_FEATURE_EXT_CAT_ERR_TYPE); + /* + * The uncorrectable local error notification opt-in was added in + * GuC v70.38.0, which maps to compatibility version v1.18.0. + */ + if (GUC_SUBMIT_VER(guc) >= MAKE_GUC_VER(1, 18, 0) && + guc_to_gt(guc)->info.has_uncorrectable_error_reporting) + klvs[count++] = + PREP_GUC_KLV_TAG(OPT_IN_FEATURE_UNCORRECTABLE_LOCAL_ERROR_NOTIFICATION); + if (supports_dynamic_ics(guc)) klvs[count++] = PREP_GUC_KLV_TAG(OPT_IN_FEATURE_DYNAMIC_INHIBIT_CONTEXT_SWITCH); @@ -1846,6 +1864,58 @@ bool xe_guc_using_main_gamctrl_queues(struct xe_guc *guc) return GT_VER(gt) >= 35; } +bool xe_guc_has_paging_engine(struct xe_guc *guc) +{ + struct xe_gt *gt = guc_to_gt(guc); + struct xe_device *xe = gt_to_xe(gt); + + /* + * On newer platforms the GuC now has a dedicated engine class for the + * special PAGING engine, which is the driver reserved BCS engine used + * for KMD paging/binding operations. GuC requires KMD to refer to this + * using the special PAGING engine class. Note that there is no new hw + * engine here, this is purely a sw view in the GuC itself, which we + * need to respect. + */ + + if (IS_SRIOV_VF(xe)) + return xe_gt_sriov_vf_paging_engines(gt); + + return xe->info.platform >= XE_NOVALAKE_S && + GUC_FIRMWARE_VER_AT_LEAST(guc, 70, 69, 0); +} + +/** + * xe_hwe_guc_logical_instance - Get the GuC-aligned logical instance of a + * hardware engine. + * @hwe: Hardware engine. + * + * For GuC backend usage, we should no longer use the raw logical instance + * directly. This helper must be used to retrieve the logical instance of the + * hardware engine, taking care of any necessary adjustments (such as the GuC + * PAGING engine mapping). This is assumed to be used in conjunction with the + * GuC engine class. + * + * Return: Logical instance, taking into account for stuff like GuC PAGING + * engine mapping. + */ +u16 xe_hwe_guc_logical_instance(struct xe_hw_engine *hwe) +{ + struct xe_gt *gt = hwe->gt; + + if (xe_guc_has_paging_engine(&hwe->gt->uc.guc) && + xe_gt_is_usm_hwe(gt, hwe)) { + int shift = gt->usm.paging_hwe0->logical_instance; + + xe_gt_assert(gt, shift <= hwe->logical_instance); + + /* GUC_PAGING_CLASS:guc_logical_instance */ + return hwe->logical_instance - shift; + } + + return hwe->logical_instance; +} + #if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST) #include "tests/xe_guc_g2g_test.c" #endif diff --git a/drivers/gpu/drm/xe/xe_guc.h b/drivers/gpu/drm/xe/xe_guc.h index 02514914f404..61e3ee19a59b 100644 --- a/drivers/gpu/drm/xe/xe_guc.h +++ b/drivers/gpu/drm/xe/xe_guc.h @@ -62,31 +62,14 @@ void xe_guc_stop(struct xe_guc *guc); int xe_guc_start(struct xe_guc *guc); void xe_guc_declare_wedged(struct xe_guc *guc); bool xe_guc_using_main_gamctrl_queues(struct xe_guc *guc); +bool xe_guc_has_paging_engine(struct xe_guc *guc); #if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST) int xe_guc_g2g_test_notification(struct xe_guc *guc, u32 *payload, u32 len); #endif -static inline u16 xe_engine_class_to_guc_class(enum xe_engine_class class) -{ - switch (class) { - case XE_ENGINE_CLASS_RENDER: - return GUC_RENDER_CLASS; - case XE_ENGINE_CLASS_VIDEO_DECODE: - return GUC_VIDEO_CLASS; - case XE_ENGINE_CLASS_VIDEO_ENHANCE: - return GUC_VIDEOENHANCE_CLASS; - case XE_ENGINE_CLASS_COPY: - return GUC_BLITTER_CLASS; - case XE_ENGINE_CLASS_COMPUTE: - return GUC_COMPUTE_CLASS; - case XE_ENGINE_CLASS_OTHER: - return GUC_GSC_OTHER_CLASS; - default: - XE_WARN_ON(class); - return -1; - } -} +u16 xe_hwe_to_guc_class(struct xe_hw_engine *hwe); +u16 xe_hwe_guc_logical_instance(struct xe_hw_engine *hwe); static inline struct xe_gt *guc_to_gt(struct xe_guc *guc) { diff --git a/drivers/gpu/drm/xe/xe_guc_ads.c b/drivers/gpu/drm/xe/xe_guc_ads.c index c98454545a85..f0ac00586d3a 100644 --- a/drivers/gpu/drm/xe/xe_guc_ads.c +++ b/drivers/gpu/drm/xe/xe_guc_ads.c @@ -16,6 +16,7 @@ #include "regs/xe_gt_regs.h" #include "regs/xe_guc_regs.h" #include "xe_bo.h" +#include "xe_configfs.h" #include "xe_gt.h" #include "xe_gt_ccs_mode.h" #include "xe_gt_mcr.h" @@ -251,16 +252,42 @@ static size_t calculate_regset_size(struct xe_gt *gt) return count * sizeof(struct guc_mmio_reg); } -static u32 engine_enable_mask(struct xe_gt *gt, enum xe_engine_class class) +static inline enum xe_engine_class guc_class_to_engine_class(u16 guc_class) +{ + switch (guc_class) { + case GUC_RENDER_CLASS: + return XE_ENGINE_CLASS_RENDER; + case GUC_VIDEO_CLASS: + return XE_ENGINE_CLASS_VIDEO_DECODE; + case GUC_VIDEOENHANCE_CLASS: + return XE_ENGINE_CLASS_VIDEO_ENHANCE; + case GUC_BLITTER_CLASS: + case GUC_PAGING_CLASS: + return XE_ENGINE_CLASS_COPY; + case GUC_COMPUTE_CLASS: + return XE_ENGINE_CLASS_COMPUTE; + case GUC_GSC_OTHER_CLASS: + return XE_ENGINE_CLASS_OTHER; + default: + XE_WARN_ON(guc_class); + return -1; + } +} + +static u32 engine_enable_mask(struct xe_gt *gt, u16 guc_class) { struct xe_hw_engine *hwe; enum xe_hw_engine_id id; u32 mask = 0; for_each_hw_engine(hwe, gt, id) - if (hwe->class == class) + if (xe_hwe_to_guc_class(hwe) == guc_class) mask |= BIT(hwe->instance); + /* We expect at most one paging engine per GuC instance, for now */ + if (guc_class == GUC_PAGING_CLASS) + xe_gt_assert(gt, !mask || is_power_of_2(mask)); + return mask; } @@ -268,10 +295,13 @@ static size_t calculate_golden_lrc_size(struct xe_guc_ads *ads) { struct xe_gt *gt = ads_to_gt(ads); size_t total_size = 0, alloc_size, real_size; - int class; + u16 guc_class; + + for (guc_class = 0; guc_class <= GUC_LAST_ENGINE_CLASS; ++guc_class) { + enum xe_engine_class class = + guc_class_to_engine_class(guc_class); - for (class = 0; class < XE_ENGINE_CLASS_MAX; ++class) { - if (!engine_enable_mask(gt, class)) + if (!engine_enable_mask(gt, guc_class)) continue; real_size = xe_gt_lrc_size(gt, class); @@ -364,6 +394,28 @@ static void guc_waklv_init(struct xe_guc_ads *ads) guc_waklv_enable(ads, NULL, 0, &offset, &remain, GUC_WA_KLV_CLR_CS_INDIRECT_RING_STATE_IF_IDLE_AT_CTX_REG); + if (XE_GT_WA(gt, 22022079272) && GUC_FIRMWARE_VER_AT_LEAST(>->uc.guc, 70, 62)) + guc_waklv_enable(ads, NULL, 0, &offset, &remain, GUC_WA_KLV_REMAP_RANGED_TLB_INV); + + /* The GuC does not enable the sem_tok_64 feature on NVL-S */ + if (XE_GT_WA(gt, 16029897822) && gt_to_xe(gt)->info.platform != XE_NOVALAKE_S && + GUC_FIRMWARE_VER_AT_LEAST(>->uc.guc, 70, 69)) + guc_waklv_enable(ads, NULL, 0, &offset, &remain, + GUC_WA_KLV_IGNORE_MMIO_READ_SEM_TOKEN_64); + + /* + * On GuC firmware 70.66 and above, use the Feature KLV (shared with the + * WA KLV buffer); older firmware uses GUC_CTL_DISABLE_MULTI_QUEUE in + * the init params instead. + */ + if (!xe_configfs_get_enable_multi_queue(to_pci_dev(gt_to_xe(gt)->drm.dev)) && + GUC_FIRMWARE_VER_AT_LEAST(>->uc.guc, 70, 66)) { + u32 data = 1; + + guc_waklv_enable(ads, &data, 1, &offset, &remain, + GUC_FEATURE_KLV_DISABLE_MULTI_QUEUE); + } + size = guc_ads_waklv_size(ads) - remain; if (!size) return; @@ -463,20 +515,36 @@ static void fill_engine_enable_masks(struct xe_gt *gt, struct iosys_map *info_map) { struct xe_device *xe = gt_to_xe(gt); + u16 guc_class; + + for (guc_class = 0; guc_class <= GUC_LAST_ENGINE_CLASS; ++guc_class) + info_map_write(xe, info_map, engine_enabled_masks[guc_class], + engine_enable_mask(gt, guc_class)); +} - info_map_write(xe, info_map, engine_enabled_masks[GUC_RENDER_CLASS], - engine_enable_mask(gt, XE_ENGINE_CLASS_RENDER)); - info_map_write(xe, info_map, engine_enabled_masks[GUC_BLITTER_CLASS], - engine_enable_mask(gt, XE_ENGINE_CLASS_COPY)); - info_map_write(xe, info_map, engine_enabled_masks[GUC_VIDEO_CLASS], - engine_enable_mask(gt, XE_ENGINE_CLASS_VIDEO_DECODE)); - info_map_write(xe, info_map, - engine_enabled_masks[GUC_VIDEOENHANCE_CLASS], - engine_enable_mask(gt, XE_ENGINE_CLASS_VIDEO_ENHANCE)); - info_map_write(xe, info_map, engine_enabled_masks[GUC_COMPUTE_CLASS], - engine_enable_mask(gt, XE_ENGINE_CLASS_COMPUTE)); - info_map_write(xe, info_map, engine_enabled_masks[GUC_GSC_OTHER_CLASS], - engine_enable_mask(gt, XE_ENGINE_CLASS_OTHER)); +u16 xe_hwe_to_guc_class(struct xe_hw_engine *hwe) +{ + if (xe_guc_has_paging_engine(&hwe->gt->uc.guc) && + xe_gt_is_usm_hwe(hwe->gt, hwe)) + return GUC_PAGING_CLASS; + + switch (hwe->class) { + case XE_ENGINE_CLASS_RENDER: + return GUC_RENDER_CLASS; + case XE_ENGINE_CLASS_VIDEO_DECODE: + return GUC_VIDEO_CLASS; + case XE_ENGINE_CLASS_VIDEO_ENHANCE: + return GUC_VIDEOENHANCE_CLASS; + case XE_ENGINE_CLASS_COPY: + return GUC_BLITTER_CLASS; + case XE_ENGINE_CLASS_COMPUTE: + return GUC_COMPUTE_CLASS; + case XE_ENGINE_CLASS_OTHER: + return GUC_GSC_OTHER_CLASS; + default: + XE_WARN_ON(hwe->class); + return -1; + } } /* @@ -491,15 +559,14 @@ static void guc_golden_lrc_init(struct xe_guc_ads *ads) offsetof(struct __guc_ads_blob, system_info)); size_t alloc_size, real_size; u32 addr_ggtt, offset; - int class; + u16 guc_class; offset = guc_ads_golden_lrc_offset(ads); addr_ggtt = xe_bo_ggtt_addr(ads->bo) + offset; - for (class = 0; class < XE_ENGINE_CLASS_MAX; ++class) { - u8 guc_class; - - guc_class = xe_engine_class_to_guc_class(class); + for (guc_class = 0; guc_class <= GUC_LAST_ENGINE_CLASS; ++guc_class) { + enum xe_engine_class class = + guc_class_to_engine_class(guc_class); if (!info_map_read(xe, &info_map, engine_enabled_masks[guc_class])) @@ -548,11 +615,14 @@ static void guc_mapping_table_init(struct xe_gt *gt, guc_mapping_table_init_invalid(gt, info_map); for_each_hw_engine(hwe, gt, id) { + u16 guc_logical_instance; u8 guc_class; - guc_class = xe_engine_class_to_guc_class(hwe->class); + guc_class = xe_hwe_to_guc_class(hwe); + guc_logical_instance = xe_hwe_guc_logical_instance(hwe); + info_map_write(xe, info_map, - mapping_table[guc_class][hwe->logical_instance], + mapping_table[guc_class][guc_logical_instance], hwe->instance); } } @@ -580,6 +650,9 @@ static u32 guc_get_capture_engine_mask(struct xe_gt *gt, struct iosys_map *info_ case GUC_CAPTURE_LIST_CLASS_GSC_OTHER: mask = info_map_read(xe, info_map, engine_enabled_masks[GUC_GSC_OTHER_CLASS]); break; + case GUC_CAPTURE_LIST_CLASS_PAGING: + mask = info_map_read(xe, info_map, engine_enabled_masks[GUC_PAGING_CLASS]); + break; default: mask = 0; } @@ -805,7 +878,7 @@ static void guc_mmio_reg_state_init(struct xe_guc_ads *ads) * 2. Record in the header (ads.reg_state_list) the address * location and number of entries */ - gc = xe_engine_class_to_guc_class(hwe->class); + gc = xe_hwe_to_guc_class(hwe); ads_blob_write(ads, ads.reg_state_list[gc][hwe->instance].address, addr); ads_blob_write(ads, ads.reg_state_list[gc][hwe->instance].count, count); @@ -948,14 +1021,13 @@ static void guc_golden_lrc_populate(struct xe_guc_ads *ads) offsetof(struct __guc_ads_blob, system_info)); size_t total_size = 0, alloc_size, real_size; u32 offset; - int class; + u16 guc_class; offset = guc_ads_golden_lrc_offset(ads); - for (class = 0; class < XE_ENGINE_CLASS_MAX; ++class) { - u8 guc_class; - - guc_class = xe_engine_class_to_guc_class(class); + for (guc_class = 0; guc_class <= GUC_LAST_ENGINE_CLASS; ++guc_class) { + enum xe_engine_class class = + guc_class_to_engine_class(guc_class); if (!info_map_read(xe, &info_map, engine_enabled_masks[guc_class])) diff --git a/drivers/gpu/drm/xe/xe_guc_capture.c b/drivers/gpu/drm/xe/xe_guc_capture.c index 1a019137ddf4..82df19b304e1 100644 --- a/drivers/gpu/drm/xe/xe_guc_capture.c +++ b/drivers/gpu/drm/xe/xe_guc_capture.c @@ -249,6 +249,8 @@ static const struct __guc_mmio_reg_descr_group xe_hpg_lists[] = { MAKE_REGLIST(xe_blt_inst_regs, PF, ENGINE_INSTANCE, GUC_CAPTURE_LIST_CLASS_BLITTER), MAKE_REGLIST(empty_regs_list, PF, ENGINE_CLASS, GUC_CAPTURE_LIST_CLASS_GSC_OTHER), MAKE_REGLIST(xe_lp_gsc_inst_regs, PF, ENGINE_INSTANCE, GUC_CAPTURE_LIST_CLASS_GSC_OTHER), + MAKE_REGLIST(empty_regs_list, PF, ENGINE_CLASS, GUC_CAPTURE_LIST_CLASS_PAGING), + MAKE_REGLIST(xe_blt_inst_regs, PF, ENGINE_INSTANCE, GUC_CAPTURE_LIST_CLASS_PAGING), {} }; @@ -265,6 +267,8 @@ static const struct __guc_mmio_reg_descr_group xe3p_lists[] = { MAKE_REGLIST(xe_blt_inst_regs, PF, ENGINE_INSTANCE, GUC_CAPTURE_LIST_CLASS_BLITTER), MAKE_REGLIST(empty_regs_list, PF, ENGINE_CLASS, GUC_CAPTURE_LIST_CLASS_GSC_OTHER), MAKE_REGLIST(xe_lp_gsc_inst_regs, PF, ENGINE_INSTANCE, GUC_CAPTURE_LIST_CLASS_GSC_OTHER), + MAKE_REGLIST(empty_regs_list, PF, ENGINE_CLASS, GUC_CAPTURE_LIST_CLASS_PAGING), + MAKE_REGLIST(xe_blt_inst_regs, PF, ENGINE_INSTANCE, GUC_CAPTURE_LIST_CLASS_PAGING), {} }; static const char * const capture_list_type_names[] = { @@ -279,6 +283,7 @@ static const char * const capture_engine_class_names[] = { "VideoEnhance", "Blitter", "GSC-Other", + "Paging", }; struct __guc_capture_ads_cache { @@ -440,7 +445,7 @@ static void guc_capture_alloc_steered_lists(struct xe_guc *guc) * to be extended */ for_each_hw_engine(hwe, gt, id) { - if (xe_engine_class_to_guc_capture_class(hwe->class) == + if (xe_hwe_to_guc_capture_class(hwe) == GUC_CAPTURE_LIST_CLASS_RENDER_COMPUTE) { has_rcs_ccs = true; break; @@ -772,6 +777,10 @@ size_t xe_guc_capture_ads_input_worst_size(struct xe_guc *guc) total_size = PAGE_SIZE; /* Pad a page in front for empty lists */ for (i = 0; i < GUC_CAPTURE_LIST_INDEX_MAX; i++) { for (j = 0; j < GUC_CAPTURE_LIST_CLASS_MAX; j++) { + if (!xe_guc_has_paging_engine(guc) && + j == GUC_CAPTURE_LIST_CLASS_PAGING) + continue; + if (xe_guc_capture_getlistsize(guc, i, GUC_STATE_CAPTURE_TYPE_ENGINE_CLASS, j, &class_size) < 0) @@ -818,7 +827,7 @@ static int guc_capture_output_size_est(struct xe_guc *guc) for_each_hw_engine(hwe, gt, id) { enum guc_capture_list_class_type capture_class; - capture_class = xe_engine_class_to_guc_capture_class(hwe->class); + capture_class = xe_hwe_to_guc_capture_class(hwe); capture_size += sizeof(struct guc_state_capture_group_header_t) + (3 * sizeof(struct guc_state_capture_header_t)); @@ -1626,7 +1635,7 @@ xe_engine_manual_capture(struct xe_hw_engine *hwe, struct xe_hw_engine_snapshot if (!new) return; - capture_class = xe_engine_class_to_guc_capture_class(hwe->class); + capture_class = xe_hwe_to_guc_capture_class(hwe); for (type = GUC_STATE_CAPTURE_TYPE_GLOBAL; type < GUC_STATE_CAPTURE_TYPE_MAX; type++) { struct gcap_reg_list_info *reginfo = &new->reginfo[type]; /* @@ -1668,7 +1677,7 @@ xe_engine_manual_capture(struct xe_hw_engine *hwe, struct xe_hw_engine_snapshot } } - new->eng_class = xe_engine_class_to_guc_class(hwe->class); + new->eng_class = xe_hwe_to_guc_class(hwe); new->eng_inst = hwe->instance; new->guc_id = guc_id; new->lrca = lrca; @@ -1832,7 +1841,7 @@ void xe_engine_snapshot_print(struct xe_hw_engine_snapshot *snapshot, struct drm xe_gt_assert(gt, snapshot->hwe); - capture_class = xe_engine_class_to_guc_capture_class(snapshot->hwe->class); + capture_class = xe_hwe_to_guc_capture_class(snapshot->hwe); drm_printf(p, "%s (physical), logical instance=%d\n", snapshot->name ? snapshot->name : "", @@ -1904,7 +1913,7 @@ xe_guc_capture_get_matching_and_lock(struct xe_exec_queue *q) for_each_hw_engine(hwe, q->gt, id) { if (hwe != q->hwe) continue; - guc_class = xe_engine_class_to_guc_class(hwe->class); + guc_class = xe_hwe_to_guc_class(hwe); break; } diff --git a/drivers/gpu/drm/xe/xe_guc_capture.h b/drivers/gpu/drm/xe/xe_guc_capture.h index dca97d52b192..fcd4f1298536 100644 --- a/drivers/gpu/drm/xe/xe_guc_capture.h +++ b/drivers/gpu/drm/xe/xe_guc_capture.h @@ -28,6 +28,8 @@ static inline enum guc_capture_list_class_type xe_guc_class_to_capture_class(u16 case GUC_VIDEOENHANCE_CLASS: case GUC_BLITTER_CLASS: return class; + case GUC_PAGING_CLASS: + return GUC_CAPTURE_LIST_CLASS_PAGING; default: XE_WARN_ON(class); return GUC_CAPTURE_LIST_CLASS_MAX; @@ -35,9 +37,9 @@ static inline enum guc_capture_list_class_type xe_guc_class_to_capture_class(u16 } static inline enum guc_capture_list_class_type -xe_engine_class_to_guc_capture_class(enum xe_engine_class class) +xe_hwe_to_guc_capture_class(struct xe_hw_engine *hwe) { - return xe_guc_class_to_capture_class(xe_engine_class_to_guc_class(class)); + return xe_guc_class_to_capture_class(xe_hwe_to_guc_class(hwe)); } void xe_guc_capture_process(struct xe_guc *guc); diff --git a/drivers/gpu/drm/xe/xe_guc_ct.c b/drivers/gpu/drm/xe/xe_guc_ct.c index 21e0dad9a481..fe70c0fd85c5 100644 --- a/drivers/gpu/drm/xe/xe_guc_ct.c +++ b/drivers/gpu/drm/xe/xe_guc_ct.c @@ -1065,6 +1065,11 @@ static int __guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, xe_gt_assert(gt, g2h_len || !num_g2h); lockdep_assert_held(&ct->lock); + if (xe_device_wedged(ct_to_xe(ct))) { + ret = -ENOTRECOVERABLE; + goto out; + } + if (unlikely(ct->ctbs.h2g.info.broken)) { ret = -EPIPE; goto out; @@ -1236,6 +1241,36 @@ static int guc_ct_send(struct xe_guc_ct *ct, const u32 *action, u32 len, return ret; } +/** + * xe_guc_ct_send - Send an HXG message to the GuC over CT + * @ct: the &xe_guc_ct + * @action: dword array with the HXG message (can't be NULL) + * @len: length of the HXG message in dwords (can't be 0) + * @g2h_len: G2H response space to reserve in dwords, or 0 + * @num_g2h: number of G2H messages expected, or 0 + * + * Return codes from the non-blocking send helpers are: + * + * * -ENOTRECOVERABLE: the xe device is wedged. Stop submitting new GuC work; the + * request cannot make progress until the device is recovered. + * * -EPIPE: the H2G CTB is marked broken. The channel stays unusable until the + * CT is restarted, which clears the broken flag. + * * -ENODEV: the CT channel is disabled, messages not expected in this state. + * Don't retry until it is enabled again. + * * -ECANCELED: the CT channel is stopped or a GT recovery is pending; the + * message was dropped. Often benign. Cancel-tolerant callers (e.g. TLB + * invalidations, GuC submission) rely on the stop/start flow to recover; + * others should retry once the CT is re-enabled or the reset/recovery + * completes. + * * -EDEADLK: no CTB room and the wait for space timed out. The send helpers + * have already requested an async GT reset before returning this error. + * + * -ENOMEM may also be returned if an internal allocation fails; the blocking + * xe_guc_ct_send_recv() path retries that allocation. -EBUSY and + * -EAGAIN are internal flow-control results handled by the send helpers. + * + * Return: 0 on success, or a negative error code on failure. + */ int xe_guc_ct_send(struct xe_guc_ct *ct, const u32 *action, u32 len, u32 g2h_len, u32 num_g2h) { @@ -1388,7 +1423,7 @@ wait_again: if (g2h_fence.fail) { if (g2h_fence.cancel) { xe_gt_dbg(gt, "H2G request %#x canceled!\n", action[0]); - ret = -ECANCELED; + ret = xe_device_wedged(ct_to_xe(ct)) ? -ENOTRECOVERABLE : -ECANCELED; goto unlock; } xe_gt_err(gt, "H2G request %#x failed: error %#x hint %#x\n", @@ -1661,6 +1696,9 @@ static int process_g2h_msg(struct xe_guc_ct *ct, u32 *msg, u32 len) ret = xe_guc_exec_queue_memory_cat_error_handler(guc, payload, adj_len); break; + case XE_GUC_ACTION_NOTIFY_UNCORRECTABLE_LOCAL_ERROR: + ret = xe_guc_uncorrectable_error_handler(guc, payload, adj_len); + break; case XE_GUC_ACTION_REPORT_PAGE_FAULT_REQ_DESC: ret = xe_guc_pagefault_handler(guc, payload, adj_len); break; @@ -1724,6 +1762,9 @@ static int g2h_read(struct xe_guc_ct *ct, u32 *msg, bool fast_path) xe_gt_assert(gt, xe_guc_ct_initialized(ct)); lockdep_assert_held(&ct->fast_lock); + if (xe_device_wedged(xe)) + return -ENOTRECOVERABLE; + if (ct->state == XE_GUC_CT_STATE_DISABLED) return -ENODEV; diff --git a/drivers/gpu/drm/xe/xe_guc_engine_activity.c b/drivers/gpu/drm/xe/xe_guc_engine_activity.c index 2b99c1ebdd58..a782be57caad 100644 --- a/drivers/gpu/drm/xe/xe_guc_engine_activity.c +++ b/drivers/gpu/drm/xe/xe_guc_engine_activity.c @@ -27,7 +27,8 @@ static struct iosys_map engine_activity_map(struct xe_guc *guc, struct xe_hw_eng { struct xe_guc_engine_activity *engine_activity = &guc->engine_activity; struct engine_activity_buffer *buffer; - u16 guc_class = xe_engine_class_to_guc_class(hwe->class); + u16 guc_class = xe_hwe_to_guc_class(hwe); + u16 guc_logical_instance = xe_hwe_guc_logical_instance(hwe); size_t offset; if (engine_activity->num_functions) { @@ -39,7 +40,7 @@ static struct iosys_map engine_activity_map(struct xe_guc *guc, struct xe_hw_eng } offset += offsetof(struct guc_engine_activity_data, - engine_activity[guc_class][hwe->logical_instance]); + engine_activity[guc_class][guc_logical_instance]); return IOSYS_MAP_INIT_OFFSET(&buffer->activity_bo->vmap, offset); } @@ -150,9 +151,10 @@ static struct engine_activity *hw_engine_to_engine_activity(struct xe_hw_engine { struct xe_guc *guc = &hwe->gt->uc.guc; struct engine_activity_group *eag = &guc->engine_activity.eag[index]; - u16 guc_class = xe_engine_class_to_guc_class(hwe->class); + u16 guc_class = xe_hwe_to_guc_class(hwe); + u16 guc_logical_instance = xe_hwe_guc_logical_instance(hwe); - return &eag->engine[guc_class][hwe->logical_instance]; + return &eag->engine[guc_class][guc_logical_instance]; } static u64 cpu_ns_to_guc_tsc_tick(ktime_t ns, u32 freq) @@ -473,7 +475,7 @@ void xe_guc_engine_activity_enable_stats(struct xe_guc *guc) ret = enable_engine_activity_stats(guc); if (ret) - xe_gt_err(guc_to_gt(guc), "failed to enable activity stats%d\n", ret); + xe_gt_err(guc_to_gt(guc), "failed to enable activity stats: %pe\n", ERR_PTR(ret)); else engine_activity_set_cpu_ts(guc, 0); } diff --git a/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h b/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h index e5e53b421f29..acdc24d1a6bd 100644 --- a/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h +++ b/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h @@ -10,6 +10,7 @@ #include <linux/workqueue.h> #include "xe_gpu_scheduler_types.h" +#include "xe_hw_fence_types.h" struct dma_fence; struct xe_exec_queue; @@ -24,6 +25,10 @@ struct xe_guc_exec_queue { struct rcu_head rcu; /** @sched: GPU scheduler for this xe_exec_queue */ struct xe_gpu_scheduler sched; + /** + * @name: Scheduler timeline name, kept with @sched until RCU free. + */ + char name[MAX_FENCE_NAME_LEN]; /** @entity: Scheduler entity for this xe_exec_queue */ struct xe_sched_entity entity; /** @@ -50,6 +55,13 @@ struct xe_guc_exec_queue { /** @suspend_pending: a suspend of the exec_queue is pending */ bool suspend_pending; /** + * @suspend_count: Reference count of active suspend requests. The + * exec_queue remains suspended while this is non-zero, allowing + * multiple concurrent callers to independently hold a suspend without + * prematurely re-enabling the queue. Protected by @sched.msg_lock. + */ + int suspend_count; + /** * @needs_cleanup: Needs a cleanup message during VF post migration * recovery. */ diff --git a/drivers/gpu/drm/xe/xe_guc_fwif.h b/drivers/gpu/drm/xe/xe_guc_fwif.h index 3fbda4798cff..971b850f2136 100644 --- a/drivers/gpu/drm/xe/xe_guc_fwif.h +++ b/drivers/gpu/drm/xe/xe_guc_fwif.h @@ -68,6 +68,7 @@ struct guc_update_exec_queue_policy { #define GUC_CTL_MAIN_GAMCTRL_QUEUES BIT(9) #define GUC_CTL_DISABLE_SCHEDULER BIT(14) #define GUC_CTL_ENABLE_L2FLUSH_OPT BIT(15) +#define GUC_CTL_DISABLE_MULTI_QUEUE BIT(24) #define GUC_CTL_DEBUG 3 #define GUC_LOG_VERBOSITY REG_GENMASK(1, 0) diff --git a/drivers/gpu/drm/xe/xe_guc_klv_helpers.c b/drivers/gpu/drm/xe/xe_guc_klv_helpers.c index 97600edda837..dc8612d761e3 100644 --- a/drivers/gpu/drm/xe/xe_guc_klv_helpers.c +++ b/drivers/gpu/drm/xe/xe_guc_klv_helpers.c @@ -4,14 +4,27 @@ */ #include <linux/bitfield.h> +#include <kunit/static_stub.h> #include <drm/drm_print.h> #include "abi/guc_klvs_abi.h" +#include "abi/xe_driver_klvs_abi.h" #include "xe_guc_klv_helpers.h" #include "xe_guc_klv_thresholds_set.h" #define make_u64(hi, lo) ((u64)((u64)(u32)(hi) << 32 | (u32)(lo))) +static bool is_group_key(u16 key) +{ + KUNIT_STATIC_STUB_REDIRECT(is_group_key, key); + return false; +} + +static bool is_reserved_key(u16 key) +{ + return in_range(key, GUC_KLV_RESERVED_RANGE_START, GUC_KLV_RESERVED_RANGE_LEN); +} + /** * xe_guc_klv_key_to_string - Convert KLV key into friendly name. * @key: the `GuC KLV`_ key @@ -24,6 +37,8 @@ const char *xe_guc_klv_key_to_string(u16 key) /* GuC Global Config KLVs */ case GUC_KLV_GLOBAL_CFG_GROUP_SCHEDULING_AVAILABLE_KEY: return "group_scheduling_available"; + case GUC_KLV_GLOBAL_CFG_NUM_PAGING_ENGINE_INSTANCES_KEY: + return "num_paging_engine_instances"; /* VGT POLICY keys */ case GUC_KLV_VGT_POLICY_SCHED_IF_IDLE_KEY: return "sched_if_idle"; @@ -71,12 +86,64 @@ const char *xe_guc_klv_key_to_string(u16 key) MAKE_XE_GUC_KLV_THRESHOLDS_SET(define_threshold_key_to_string_case) #undef define_threshold_key_to_string_case + /* driver KLVs */ + case MIGRATION_KLV_DEVICE_DEVID_KEY: + return "migration_devid"; + case MIGRATION_KLV_DEVICE_REVID_KEY: + return "migration_revid"; + default: + if (is_reserved_key(key)) + return "(reserved)"; return "(unknown)"; } } /** + * xe_guc_klv_print_one() - Print single `GuC KLV`_. + * @key: KLV key + * @len: KLV length (in u32 dwords) of the KLV @value + * @value: KLV value (as array of @len u32 dwords) + * @p: the &drm_printer + * + * The buffer may contain more than one KLV. + */ +void xe_guc_klv_print_one(u16 key, u16 len, const u32 *value, struct drm_printer *p) +{ + const char *name = xe_guc_klv_key_to_string(key); + + if (is_group_key(key)) { + struct drm_printer gp = drm_line_printer(p, name, 0); + + drm_printf(p, "{ key %#06x : group %u dwords } # %s\n", + key, len, name); + + /* print group recursively */ + xe_guc_klv_print(value, len, &gp); + return; + } + + switch (len) { + case 0: + drm_printf(p, "{ key %#06x : no value } # %s\n", key, name); + break; + case 1: + drm_printf(p, "{ key %#06x : 32b value %u } # %s\n", + key, value[0], name); + break; + case 2: + drm_printf(p, "{ key %#06x : 64b value %#llx } # %s\n", + key, make_u64(value[1], value[0]), name); + break; + default: + drm_printf(p, "{ key %#06x : %zu bytes %*ph } # %s\n", + key, len * sizeof(u32), (int)(len * sizeof(u32)), + value, name); + break; + } +} + +/** * xe_guc_klv_print - Print content of the buffer with `GuC KLV`_. * @klvs: the buffer with KLVs * @num_dwords: number of dwords (u32) available in the buffer @@ -101,26 +168,7 @@ void xe_guc_klv_print(const u32 *klvs, u32 num_dwords, struct drm_printer *p) return; } - switch (len) { - case 0: - drm_printf(p, "{ key %#06x : no value } # %s\n", - key, xe_guc_klv_key_to_string(key)); - break; - case 1: - drm_printf(p, "{ key %#06x : 32b value %u } # %s\n", - key, klvs[0], xe_guc_klv_key_to_string(key)); - break; - case 2: - drm_printf(p, "{ key %#06x : 64b value %#llx } # %s\n", - key, make_u64(klvs[1], klvs[0]), - xe_guc_klv_key_to_string(key)); - break; - default: - drm_printf(p, "{ key %#06x : %zu bytes %*ph } # %s\n", - key, len * sizeof(u32), (int)(len * sizeof(u32)), - klvs, xe_guc_klv_key_to_string(key)); - break; - } + xe_guc_klv_print_one(key, len, klvs, p); klvs += len; num_dwords -= len; @@ -155,3 +203,175 @@ int xe_guc_klv_count(const u32 *klvs, u32 num_dwords) return num_dwords ? -ENODATA : num_klvs; } + +static size_t to_num_bytes(u16 dwords) +{ + return dwords * sizeof(u32); +} + +static u16 to_num_dwords(size_t size) +{ + return round_up(size, sizeof(u32)) / sizeof(u32); +} + +/** + * xe_guc_klv_encode_u32() - Encode 32-bit value as KLV. + * @klvs: the buffer where to place KLV + * @avail: number of dwords (u32) available in the buffer + * @key: key to be used + * @value: value to be encoded + * + * Return: pointer to the buffer location past the encoded KLV or + * an ERR_PTR if there was no space to encode the KLV. + */ +u32 *xe_guc_klv_encode_u32(u32 *klvs, u32 avail, u16 key, u32 value) +{ + u16 len = to_num_dwords(sizeof(u32)); + + if (IS_ERR(klvs)) + return klvs; + + if (avail < GUC_KLV_LEN_MIN + len) + return ERR_PTR(-ENOSPC); + + *klvs++ = PREP_GUC_KLV(key, len); + *klvs++ = value; + return klvs; +} + +/** + * xe_guc_klv_encode_u64() - Encode 64-bit value as KLV. + * @klvs: the buffer where to place KLV + * @avail: number of dwords (u32) available in the buffer + * @key: key to be used + * @value: value to be encoded + * + * Return: pointer to the buffer location past the encoded KLV or + * an ERR_PTR if there was no space to encode the KLV. + */ +u32 *xe_guc_klv_encode_u64(u32 *klvs, u32 avail, u16 key, u64 value) +{ + u16 len = to_num_dwords(sizeof(u64)); + + if (IS_ERR(klvs)) + return klvs; + + if (avail < GUC_KLV_LEN_MIN + len) + return ERR_PTR(-ENOSPC); + + *klvs++ = PREP_GUC_KLV(key, len); + *klvs++ = lower_32_bits(value); + *klvs++ = upper_32_bits(value); + return klvs; +} + +/** + * xe_guc_klv_encode_string() - Encode string as KLV. + * @klvs: the buffer where to place KLV + * @avail: number of dwords (u32) available in the buffer + * @key: key to be used + * @s: string to be encoded + * + * Return: pointer to the buffer location past the encoded KLV or + * an ERR_PTR if there was no space to encode the KLV. + */ +u32 *xe_guc_klv_encode_string(u32 *klvs, u32 avail, u16 key, const char *s) +{ + size_t longest = to_num_bytes(FIELD_MAX(GUC_KLV_0_LEN)); + size_t size = strnlen(s, longest) + 1; /* \0 */ + u16 len = to_num_dwords(size); + + if (IS_ERR(klvs)) + return klvs; + + if (size > longest) + return ERR_PTR(-E2BIG); + + if (avail < GUC_KLV_LEN_MIN + len) + return ERR_PTR(-ENOSPC); + + *klvs++ = PREP_GUC_KLV(key, len); + strscpy_pad((void *)klvs, s, to_num_bytes(len)); + return klvs + len; +} + +/** + * xe_guc_klv_encode_object() - Encode object using custom encoder as single KLV. + * @klvs: the buffer where to place KLV + * @avail: number of dwords (u32) available in the buffer + * @key: key to be used + * @obj: opaque object pointer + * @encoder: function pointer to the custom encoder + * + * Return: pointer to the buffer location past the encoded KLV or + * an ERR_PTR if there was no space to encode the KLV. + */ +u32 *xe_guc_klv_encode_object(u32 *klvs, u32 avail, u16 key, const void *obj, + u32 *(*encoder)(u32 *klvs, u32 avail, const void *obj)) +{ + u32 *end; + + if (IS_ERR(klvs)) + return klvs; + + if (avail < GUC_KLV_LEN_MIN) + return ERR_PTR(-ENOSPC); + + if (avail > GUC_KLV_LEN_MIN + FIELD_MAX(GUC_KLV_0_LEN)) + avail = GUC_KLV_LEN_MIN + FIELD_MAX(GUC_KLV_0_LEN); + + end = encoder(klvs + GUC_KLV_LEN_MIN, avail - GUC_KLV_LEN_MIN, obj); + if (IS_ERR(end)) + return end; + + if (WARN_ON(end < klvs + GUC_KLV_LEN_MIN)) + return ERR_PTR(-EPIPE); + + if (WARN_ON(end > klvs + avail)) + return ERR_PTR(-EFBIG); + + *klvs = PREP_GUC_KLV(key, end - (klvs + GUC_KLV_LEN_MIN)); + return end; +} + +/** + * xe_guc_klv_parser() - Parse and decode stream of KLVs. + * @klvs: the buffer with KLVs + * @num_dwords: number of dwords (u32) available in the buffer + * @obj: opaque pointer to be used by the @decoder function + * @decoder: pointer to the decoder function + * + * Return: The sum of all results returned by the decoder or + * an -errno on decoder or buffer failure. + */ +int xe_guc_klv_parser(const u32 *klvs, u32 num_dwords, void *obj, + int (*decoder)(void *obj, u16 key, u16 len, const u32 *value)) +{ + int total = 0; + int ret; + + while (num_dwords >= GUC_KLV_LEN_MIN) { + u16 key = FIELD_GET(GUC_KLV_0_KEY, klvs[0]); + u16 len = FIELD_GET(GUC_KLV_0_LEN, klvs[0]); + + klvs += GUC_KLV_LEN_MIN; + num_dwords -= GUC_KLV_LEN_MIN; + + if (num_dwords < len) + return -ENODATA; + + ret = decoder(obj, key, len, klvs); + if (ret < 0) + return ret; + total += ret; + + klvs += len; + num_dwords -= len; + } + + return total; +} + +#if IS_BUILTIN(CONFIG_DRM_XE_KUNIT_TEST) +#include "tests/xe_guc_klv_helpers_kunit.c" +#endif diff --git a/drivers/gpu/drm/xe/xe_guc_klv_helpers.h b/drivers/gpu/drm/xe/xe_guc_klv_helpers.h index c676d21c173b..cf3b29adc9d6 100644 --- a/drivers/gpu/drm/xe/xe_guc_klv_helpers.h +++ b/drivers/gpu/drm/xe/xe_guc_klv_helpers.h @@ -13,9 +13,19 @@ struct drm_printer; const char *xe_guc_klv_key_to_string(u16 key); +void xe_guc_klv_print_one(u16 key, u16 len, const u32 *value, struct drm_printer *p); void xe_guc_klv_print(const u32 *klvs, u32 num_dwords, struct drm_printer *p); int xe_guc_klv_count(const u32 *klvs, u32 num_dwords); +u32 *xe_guc_klv_encode_u32(u32 *klvs, u32 avail, u16 key, u32 value); +u32 *xe_guc_klv_encode_u64(u32 *klvs, u32 avail, u16 key, u64 value); +u32 *xe_guc_klv_encode_string(u32 *klvs, u32 avail, u16 key, const char *s); +u32 *xe_guc_klv_encode_object(u32 *klvs, u32 avail, u16 key, const void *obj, + u32 *(*encoder)(u32 *klvs, u32 avail, const void *obj)); + +int xe_guc_klv_parser(const u32 *klvs, u32 num_dwords, void *obj, + int (*decoder)(void *obj, u16 key, u16 len, const u32 *value)); + /** * PREP_GUC_KLV - Prepare KLV header value based on provided key and len. * @key: KLV key diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c index 12416bfa3255..8aaed4fd13ea 100644 --- a/drivers/gpu/drm/xe/xe_guc_submit.c +++ b/drivers/gpu/drm/xe/xe_guc_submit.c @@ -10,6 +10,7 @@ #include <linux/circ_buf.h> #include <linux/dma-fence-array.h> +#include <drm/drm_drv.h> #include <drm/drm_managed.h> #include "abi/guc_actions_abi.h" @@ -37,6 +38,7 @@ #include "xe_macros.h" #include "xe_map.h" #include "xe_mocs.h" +#include "xe_module.h" #include "xe_pm.h" #include "xe_ring_ops_types.h" #include "xe_sched_job.h" @@ -232,17 +234,9 @@ static bool exec_queue_killed_or_banned_or_wedged(struct xe_exec_queue *q) static void guc_submit_sw_fini(struct drm_device *drm, void *arg) { struct xe_guc *guc = arg; - struct xe_device *xe = guc_to_xe(guc); struct xe_gt *gt = guc_to_gt(guc); - int ret; - - ret = wait_event_timeout(guc->submission_state.fini_wq, - xa_empty(&guc->submission_state.exec_queue_lookup), - HZ * 5); - - drain_workqueue(xe->destroy_wq); - xe_gt_assert(gt, ret); + xe_gt_assert(gt, xa_empty(&guc->submission_state.exec_queue_lookup)); xa_destroy(&guc->submission_state.exec_queue_lookup); } @@ -319,8 +313,6 @@ int xe_guc_submit_init(struct xe_guc *guc, unsigned int num_ids) xa_init(&guc->submission_state.exec_queue_lookup); - init_waitqueue_head(&guc->submission_state.fini_wq); - primelockdep(guc); guc->submission_state.initialized = true; @@ -411,9 +403,6 @@ static void __release_guc_id(struct xe_guc *guc, struct xe_exec_queue *q, xe_guc_id_mgr_release_locked(&guc->submission_state.idm, q->guc->id, q->width); - if (xa_empty(&guc->submission_state.exec_queue_lookup)) - wake_up(&guc->submission_state.fini_wq); - mutex_unlock(&guc->submission_state.lock); } @@ -972,6 +961,27 @@ static void __register_exec_queue(struct xe_guc *guc, xe_guc_ct_send(&guc->ct, action, ARRAY_SIZE(action), 0, 0); } +static u32 xe_hwe_guc_logical_to_submit_mask(struct xe_hw_engine *hwe, u32 logical_mask) +{ + struct xe_gt *gt = hwe->gt; + + if (xe_gt_is_usm_hwe(gt, hwe)) { + int shift = gt->usm.paging_hwe0->logical_instance; + u32 paging_logical_mask = gt->usm.paging_logical_mask; + + xe_gt_assert(gt, (logical_mask & paging_logical_mask) == logical_mask); + + /* + * Remap to GUC_PAGING_CLASS logical instance mask, if + * applicable. + */ + if (xe_guc_has_paging_engine(&hwe->gt->uc.guc)) + return logical_mask >> shift; + } + + return logical_mask; +} + static void register_exec_queue(struct xe_exec_queue *q, int ctx_type) { struct xe_guc *guc = exec_queue_to_guc(q); @@ -984,8 +994,9 @@ static void register_exec_queue(struct xe_exec_queue *q, int ctx_type) memset(&info, 0, sizeof(info)); info.context_idx = q->guc->id; - info.engine_class = xe_engine_class_to_guc_class(q->class); - info.engine_submit_mask = q->logical_mask; + info.engine_class = xe_hwe_to_guc_class(q->hwe); + info.engine_submit_mask = + xe_hwe_guc_logical_to_submit_mask(q->hwe, q->logical_mask); info.hwlrca_lo = lower_32_bits(xe_lrc_descriptor(lrc)); info.hwlrca_hi = upper_32_bits(xe_lrc_descriptor(lrc)); info.flags = CONTEXT_REGISTRATION_FLAG_KMD | @@ -1682,10 +1693,36 @@ handle_vf_resume: return DRM_GPU_SCHED_STAT_NO_HANG; } +static void guc_exec_queue_multi_queue_drop_suspend(struct xe_exec_queue *q); +static int guc_exec_queue_suspend_wait_blocking(struct xe_exec_queue *q); + static void guc_exec_queue_fini(struct xe_exec_queue *q) { struct xe_guc_exec_queue *ge = q->guc; struct xe_guc *guc = exec_queue_to_guc(q); + struct drm_device *drm = &guc_to_xe(guc)->drm; + + /* + * A secondary can leave the group while still preempt suspended (e.g. + * xe_vm_remove_compute_exec_queue() forces its preempt fence to signal, + * which suspends it). It holds one forwarded suspend reference on the + * primary, so drop it and resume the primary if it was the last member + * that had it suspended. Primaries forward to nobody, so they don't need + * this. + * + * First make sure the primary's forwarded suspend has completed. If the + * secondary was killed/reset before its preempt fence worker ran, that + * worker skips suspend_wait() (see preempt_fence_work_func()), leaving + * the primary's suspend possibly in flight. drop_suspend() runs under a + * spinlock and cannot wait, so drain it here with the uninterruptible + * blocking wait; otherwise resuming the primary in drop_suspend() could + * trip the !suspend_pending assert. + */ + if (xe_exec_queue_is_multi_queue_secondary(q)) { + if (READ_ONCE(q->guc->suspend_count)) + guc_exec_queue_suspend_wait_blocking(q); + guc_exec_queue_multi_queue_drop_suspend(q); + } if (xe_exec_queue_is_multi_queue_secondary(q)) { struct xe_exec_queue_group *group = q->multi_queue.group; @@ -1704,36 +1741,52 @@ static void guc_exec_queue_fini(struct xe_exec_queue *q) * (timeline name). */ kfree_rcu(ge, rcu); + + drm_dev_put(drm); } -static void __guc_exec_queue_destroy_async(struct work_struct *w) +static void guc_exec_queue_do_destroy(struct xe_exec_queue *q) { - struct xe_guc_exec_queue *ge = - container_of(w, struct xe_guc_exec_queue, destroy_async); - struct xe_exec_queue *q = ge->q; + struct xe_guc_exec_queue *ge = q->guc; struct xe_guc *guc = exec_queue_to_guc(q); + struct xe_device *xe = guc_to_xe(guc); + struct drm_device *drm = &xe->drm; - guard(xe_pm_runtime)(guc_to_xe(guc)); - trace_xe_exec_queue_destroy(q); + /* + * guc_exec_queue_fini() drops the queue's drm_device ref. + * Keep the device alive until the PM-runtime guard unwinds. + */ + drm_dev_get(drm); + + scoped_guard(xe_pm_runtime, xe) { + trace_xe_exec_queue_destroy(q); - /* Confirm no work left behind accessing device structures */ - cancel_delayed_work_sync(&ge->sched.base.work_tdr); + /* Confirm no work left behind accessing device structures */ + cancel_delayed_work_sync(&ge->sched.base.work_tdr); - xe_exec_queue_fini(q); + xe_exec_queue_fini(q); + } + + drm_dev_put(drm); } -static void guc_exec_queue_destroy_async(struct xe_exec_queue *q) +static void __guc_exec_queue_destroy_async(struct work_struct *w) { - struct xe_guc *guc = exec_queue_to_guc(q); - struct xe_device *xe = guc_to_xe(guc); + struct xe_guc_exec_queue *ge = + container_of(w, struct xe_guc_exec_queue, destroy_async); + + guc_exec_queue_do_destroy(ge->q); +} +static void guc_exec_queue_destroy_async(struct xe_exec_queue *q) +{ INIT_WORK(&q->guc->destroy_async, __guc_exec_queue_destroy_async); /* We must block on kernel engines so slabs are empty on driver unload */ if (q->flags & EXEC_QUEUE_FLAG_PERMANENT || exec_queue_wedged(q)) - __guc_exec_queue_destroy_async(&q->guc->destroy_async); + guc_exec_queue_do_destroy(q); else - queue_work(xe->destroy_wq, &q->guc->destroy_async); + xe_destroy_wq_queue(&q->guc->destroy_async); } static void __guc_exec_queue_destroy(struct xe_guc *guc, struct xe_exec_queue *q) @@ -1928,6 +1981,7 @@ static int guc_exec_queue_init(struct xe_exec_queue *q) { struct xe_gpu_scheduler *sched; struct xe_guc *guc = exec_queue_to_guc(q); + struct drm_device *drm = &guc_to_xe(guc)->drm; struct workqueue_struct *submit_wq = NULL; struct xe_guc_exec_queue *ge; long timeout; @@ -1939,6 +1993,8 @@ static int guc_exec_queue_init(struct xe_exec_queue *q) if (!ge) return -ENOMEM; + drm_dev_get(drm); + q->guc = ge; ge->q = q; init_rcu_head(&ge->rcu); @@ -1956,6 +2012,8 @@ static int guc_exec_queue_init(struct xe_exec_queue *q) xe_exec_queue_assign_name(q, q->guc->id); + strscpy(ge->name, q->name, sizeof(ge->name)); + /* * Use primary queue's submit_wq for all secondary queues of a * multi queue group. This serialization avoids any locking around @@ -1970,7 +2028,7 @@ static int guc_exec_queue_init(struct xe_exec_queue *q) err = xe_sched_init(&ge->sched, &drm_sched_ops, &xe_sched_ops, submit_wq, xe_lrc_ring_size() / MAX_JOB_SIZE_BYTES, 64, timeout, guc_to_gt(guc)->ordered_wq, NULL, - q->name, gt_to_xe(q->gt)->drm.dev); + ge->name, gt_to_xe(q->gt)->drm.dev); if (err) goto err_release_id; @@ -2015,6 +2073,7 @@ err_release_id: release_guc_id(guc, q); err_free: kfree(ge); + drm_dev_put(drm); return err; } @@ -2164,23 +2223,147 @@ static int guc_exec_queue_set_multi_queue_priority(struct xe_exec_queue *q, return 0; } -static int guc_exec_queue_suspend(struct xe_exec_queue *q) +/* + * Core suspend: take a suspend reference on @q and, on the first reference, + * disable its GuC context so the GPU is actually preempted. Caller must have + * ensured @q is not killed/banned/wedged. Returns true if this was the first + * suspend reference (the 0->1 transition). + */ +static bool __guc_exec_queue_suspend(struct xe_exec_queue *q) { - struct xe_gpu_scheduler *sched = &q->guc->sched; - struct xe_sched_msg *msg = q->guc->static_msgs + STATIC_MSG_SUSPEND; + struct xe_guc_exec_queue *ge = q->guc; + struct xe_gpu_scheduler *sched = &ge->sched; + struct xe_sched_msg *msg = ge->static_msgs + STATIC_MSG_SUSPEND; + bool first; - if (exec_queue_killed_or_banned_or_wedged(q)) - return -EINVAL; + xe_sched_msg_lock(sched); + first = (++ge->suspend_count == 1); + if (first) { + bool added = guc_exec_queue_try_add_msg(q, msg, SUSPEND); + + /* slot must be free at 0->1 */ + xe_gt_assert(guc_to_gt(exec_queue_to_guc(q)), added); + ge->suspend_pending = true; + } + xe_sched_msg_unlock(sched); + + return first; +} + +/* + * Core resume: drop a suspend reference on @q and, on the last reference, + * re-enable its GuC context. Returns true if this dropped the last suspend + * reference (the 1->0 transition). + */ +static bool __guc_exec_queue_resume(struct xe_exec_queue *q) +{ + struct xe_guc_exec_queue *ge = q->guc; + struct xe_gpu_scheduler *sched = &ge->sched; + struct xe_sched_msg *msg = ge->static_msgs + STATIC_MSG_RESUME; + struct xe_guc *guc = exec_queue_to_guc(q); + bool last; xe_sched_msg_lock(sched); - if (guc_exec_queue_try_add_msg(q, msg, SUSPEND)) - q->guc->suspend_pending = true; + xe_gt_assert(guc_to_gt(guc), !ge->suspend_pending); + xe_gt_assert(guc_to_gt(guc), ge->suspend_count > 0); + last = (--ge->suspend_count == 0); + if (last) { + bool added = guc_exec_queue_try_add_msg(q, msg, RESUME); + + /* slot must be free at 1->0 */ + xe_gt_assert(guc_to_gt(guc), added); + } xe_sched_msg_unlock(sched); + return last; +} + +static int guc_exec_queue_suspend(struct xe_exec_queue *q) +{ + if (exec_queue_killed_or_banned_or_wedged(q)) + return -EINVAL; + + /* + * Non-multi-queue queues and multi-queue primaries suspend themselves + * directly: their own msg_lock makes the suspend_count 0->1 transition + * and the suspend_pending update atomic, so no group level serialization + * is needed. + */ + if (!xe_exec_queue_is_multi_queue_secondary(q)) { + __guc_exec_queue_suspend(q); + return 0; + } + + /* + * A secondary's suspend is meaningless once the primary - which owns the + * group's GuC context - is gone, so fail it too. This keeps the + * secondary's effective state consistent with guc_exec_queue_reset_status(), + * which already reports the primary's killed/banned/wedged state for + * secondaries. A primary killed *after* this check is still handled at + * message-processing time, where the SUSPEND is a no-op for a killed + * context; this only covers an already-dead primary. + */ + if (exec_queue_killed_or_banned_or_wedged(xe_exec_queue_multi_queue_primary(q))) + return -EINVAL; + + /* + * A secondary doesn't interface with GuC: suspend it like any other + * queue (its own suspend_count drives its internally handled scheduler + * state) and, only on its own 0->1 transition, forward the suspend to the + * primary so the GPU is actually preempted. Hold @suspend_lock so that + * observing the secondary's transition and forwarding it to the primary + * happen atomically; this keeps the primary's refcount paired with member + * transitions even if the same secondary is suspended and resumed + * concurrently across rebind cycles. + */ + scoped_guard(spinlock, &q->multi_queue.group->suspend_lock) { + if (__guc_exec_queue_suspend(q)) + __guc_exec_queue_suspend(xe_exec_queue_multi_queue_primary(q)); + } + return 0; } -static int guc_exec_queue_suspend_wait(struct xe_exec_queue *q) +static void guc_exec_queue_suspend_timeout_ban(struct xe_exec_queue *q) +{ + struct xe_guc *guc = exec_queue_to_guc(q); + + xe_gt_warn(guc_to_gt(guc), + "Suspend fence, guc_id=%d, failed to respond, banning queue", + q->guc->id); + /* + * The GuC failed to respond to the suspend within the timeout. This is + * not recoverable for this context, so ban it and tear it down via + * cleanup rather than leave it suspended forever. __suspend_fence_signal + * clears suspend_pending and wakes any waiter. + * + * @q is the primary here; it owns the group's GuC context, so a failure + * to suspend it wedges the whole group. Ban and tear down the entire + * group in the multi-queue case. + */ + if (xe_exec_queue_is_multi_queue(q)) { + set_exec_queue_group_banned(q); + __suspend_fence_signal(q); + xe_guc_exec_queue_group_trigger_cleanup(q); + } else { + set_exec_queue_banned(q); + __suspend_fence_signal(q); + xe_guc_exec_queue_trigger_cleanup(q); + } +} + +/* + * Wait for @q's own suspend to complete: suspend_pending cleared, or the queue + * killed / GuC stopped. With @blocking, wait uninterruptibly and do not handle + * VF recovery (for callers that must complete on behalf of a possibly + * cross-process queue); otherwise wait interruptibly. + * + * Returns 0 on completion or -ETIME on timeout. Interruptible waits may also + * return -EAGAIN (VF recovery in progress, retry) or -ERESTARTSYS (aborted by a + * signal; suspend_pending may still be set, so callers must not resume() + * without re-confirming the suspend). + */ +static int guc_exec_queue_wait_suspend_done(struct xe_exec_queue *q, bool blocking) { struct xe_guc *guc = exec_queue_to_guc(q); struct xe_device *xe = guc_to_xe(guc); @@ -2196,44 +2379,146 @@ static int guc_exec_queue_suspend_wait(struct xe_exec_queue *q) xe_guc_read_stopped(guc)) retry: - if (IS_SRIOV_VF(xe)) + if (blocking) { + if (IS_SRIOV_VF(xe)) + ret = wait_event_timeout(guc->ct.wq, WAIT_COND, HZ * 5); + else + ret = wait_event_timeout(q->guc->suspend_wait, WAIT_COND, + HZ * 5); + } else if (IS_SRIOV_VF(xe)) { ret = wait_event_interruptible_timeout(guc->ct.wq, WAIT_COND || - vf_recovery(guc), - HZ * 5); - else + vf_recovery(guc), HZ * 5); + } else { ret = wait_event_interruptible_timeout(q->guc->suspend_wait, WAIT_COND, HZ * 5); + } - if (vf_recovery(guc) && !xe_device_wedged((guc_to_xe(guc)))) + if (!blocking && vf_recovery(guc) && !xe_device_wedged(xe)) return -EAGAIN; - if (!ret) { - xe_gt_warn(guc_to_gt(guc), - "Suspend fence, guc_id=%d, failed to respond", - q->guc->id); - /* XXX: Trigger GT reset? */ + if (!ret) return -ETIME; - } else if (IS_SRIOV_VF(xe) && !WAIT_COND) { + else if (!blocking && IS_SRIOV_VF(xe) && !WAIT_COND) /* Corner case on RESFIX DONE where vf_recovery() changes */ goto retry; - } #undef WAIT_COND return ret < 0 ? ret : 0; } +static int guc_exec_queue_suspend_wait_common(struct xe_exec_queue *q, bool blocking) +{ + int ret; + + /* + * A secondary's suspend rides the sched-message worker (short-circuited, + * no GuC round-trip) and so is not synchronous with + * guc_exec_queue_suspend(): its own suspend_pending may still be set + * here. Waiting on the primary alone is not sufficient - if the primary + * was already suspended, the forward is a refcount-only transition that + * queues no new primary SUSPEND and leaves the primary's suspend_pending + * clear, so the primary wait would return immediately while the + * secondary's suspend is still in flight, and a later resume() would trip + * the secondary's !suspend_pending assert. So first wait for the + * secondary's own suspend to complete, then wait on the primary. + * + * A timeout on either bans the queue (being multi-queue, that tears down + * the whole group). A secondary suspend has no real GuC round-trip, so + * its timeout is a software scheduler stall rather than a GuC fault, but + * banning is still the safe recovery: otherwise the queue is left with + * suspend_pending set and a subsequent resume() trips the !suspend_pending + * assert. + */ + if (xe_exec_queue_is_multi_queue_secondary(q)) { + ret = guc_exec_queue_wait_suspend_done(q, blocking); + if (ret == -ETIME) + guc_exec_queue_suspend_timeout_ban(q); + if (ret) + return ret; + } + + q = xe_exec_queue_multi_queue_primary(q); + ret = guc_exec_queue_wait_suspend_done(q, blocking); + if (ret == -ETIME) + guc_exec_queue_suspend_timeout_ban(q); + + return ret; +} + +static int guc_exec_queue_suspend_wait(struct xe_exec_queue *q) +{ + return guc_exec_queue_suspend_wait_common(q, false); +} + +/* + * Uninterruptible variant of guc_exec_queue_suspend_wait() for callers that + * must complete the wait on behalf of a queue possibly owned by a different + * process (e.g. cleanup/undo paths). An interruptible wait could return + * -ERESTARTSYS if the calling task is signalled, leaving that queue suspended + * forever (cross-process DoS). VF recovery is deliberately not handled (no + * -EAGAIN) since a blocking caller cannot retry. + */ +static int guc_exec_queue_suspend_wait_blocking(struct xe_exec_queue *q) +{ + return guc_exec_queue_suspend_wait_common(q, true); +} + static void guc_exec_queue_resume(struct xe_exec_queue *q) { - struct xe_gpu_scheduler *sched = &q->guc->sched; - struct xe_sched_msg *msg = q->guc->static_msgs + STATIC_MSG_RESUME; - struct xe_guc *guc = exec_queue_to_guc(q); + /* + * Non-multi-queue queues and multi-queue primaries resume themselves + * directly; their own msg_lock is sufficient. + */ + if (!xe_exec_queue_is_multi_queue_secondary(q)) { + __guc_exec_queue_resume(q); + return; + } - xe_gt_assert(guc_to_gt(guc), !q->guc->suspend_pending); + /* + * Mirror of guc_exec_queue_suspend(): resume the secondary like any + * other queue and, only on its own 1->0 transition, forward the resume + * to the primary so the primary's GuC context is re-enabled once the + * last member that suspended it resumes. @suspend_lock keeps the + * secondary transition and the primary forward atomic. + */ + scoped_guard(spinlock, &q->multi_queue.group->suspend_lock) { + if (__guc_exec_queue_resume(q)) + __guc_exec_queue_resume(xe_exec_queue_multi_queue_primary(q)); + } +} - xe_sched_msg_lock(sched); - guc_exec_queue_try_add_msg(q, msg, RESUME); - xe_sched_msg_unlock(sched); +/* + * Drop a leaving secondary's forwarded suspend reference on the primary and + * resume the primary if this was the last member that had it suspended. + * See guc_exec_queue_fini(). + */ +static void guc_exec_queue_multi_queue_drop_suspend(struct xe_exec_queue *q) +{ + scoped_guard(spinlock, &q->multi_queue.group->suspend_lock) { + struct xe_exec_queue *primary = xe_exec_queue_multi_queue_primary(q); + + /* + * A suspended secondary holds exactly one suspend reference on the + * primary (forwarded on its 0->1 transition). If it leaves while + * still suspended, release that reference so the primary is not + * kept disabled forever. + */ + if (!READ_ONCE(q->guc->suspend_count)) + break; + + if (exec_queue_killed_or_banned_or_wedged(primary)) + break; + + /* + * No suspend_wait() here (and we can't - suspend_lock is a + * spinlock). guc_exec_queue_fini() has already drained the + * primary's forwarded suspend with the blocking wait, so its + * suspend has completed (suspend_pending cleared) by the time we + * resume it here. __guc_exec_queue_resume() asserts this. + */ + __guc_exec_queue_resume(primary); + } } static bool guc_exec_queue_reset_status(struct xe_exec_queue *q) @@ -2262,6 +2547,7 @@ static const struct xe_exec_queue_ops guc_exec_queue_ops = { .set_multi_queue_priority = guc_exec_queue_set_multi_queue_priority, .suspend = guc_exec_queue_suspend, .suspend_wait = guc_exec_queue_suspend_wait, + .suspend_wait_blocking = guc_exec_queue_suspend_wait_blocking, .resume = guc_exec_queue_resume, .reset_status = guc_exec_queue_reset_status, }; @@ -3022,6 +3308,38 @@ int xe_guc_exec_queue_memory_cat_error_handler(struct xe_guc *guc, u32 *msg, return 0; } +int xe_guc_uncorrectable_error_handler(struct xe_guc *guc, u32 *msg, u32 len) +{ + struct xe_gt *gt = guc_to_gt(guc); + struct xe_exec_queue *q; + u32 guc_id; + + if (unlikely(!len || len > 1)) + return -EPROTO; + + guc_id = msg[0]; + + if (guc_id == GUC_ID_UNKNOWN) { + xe_gt_err(gt, "GuC: Uncorrectable local error with unknown GuC id\n"); + return 0; + } + + q = g2h_exec_queue_lookup(guc, guc_id); + if (unlikely(!q)) + return -EPROTO; + + xe_gt_err(gt, + "GuC: Uncorrectable local error! guc_id=%d class=%s, logical_mask=0x%x", + guc_id, xe_hw_engine_class_to_str(q->class), q->logical_mask); + + trace_xe_guc_uncorrectable_error(q); + + /* Treat the same as engine reset */ + xe_guc_exec_queue_reset_trigger_cleanup(q); + + return 0; +} + int xe_guc_exec_queue_reset_failure_handler(struct xe_guc *guc, u32 *msg, u32 len) { struct xe_gt *gt = guc_to_gt(guc); diff --git a/drivers/gpu/drm/xe/xe_guc_submit.h b/drivers/gpu/drm/xe/xe_guc_submit.h index b3839a90c142..ccade320dc69 100644 --- a/drivers/gpu/drm/xe/xe_guc_submit.h +++ b/drivers/gpu/drm/xe/xe_guc_submit.h @@ -34,6 +34,7 @@ int xe_guc_deregister_done_handler(struct xe_guc *guc, u32 *msg, u32 len); int xe_guc_exec_queue_reset_handler(struct xe_guc *guc, u32 *msg, u32 len); int xe_guc_exec_queue_memory_cat_error_handler(struct xe_guc *guc, u32 *msg, u32 len); +int xe_guc_uncorrectable_error_handler(struct xe_guc *guc, u32 *msg, u32 len); int xe_guc_exec_queue_reset_failure_handler(struct xe_guc *guc, u32 *msg, u32 len); int xe_guc_error_capture_handler(struct xe_guc *guc, u32 *msg, u32 len); int xe_guc_exec_queue_cgp_sync_done_handler(struct xe_guc *guc, u32 *msg, u32 len); diff --git a/drivers/gpu/drm/xe/xe_guc_types.h b/drivers/gpu/drm/xe/xe_guc_types.h index c7b9642b41ba..31a2acb63ac3 100644 --- a/drivers/gpu/drm/xe/xe_guc_types.h +++ b/drivers/gpu/drm/xe/xe_guc_types.h @@ -100,8 +100,6 @@ struct xe_guc { * even initialized - before that not even the lock is valid */ bool initialized; - /** @submission_state.fini_wq: submit fini wait queue */ - wait_queue_head_t fini_wq; } submission_state; /** @hwconfig: Hardware config state */ diff --git a/drivers/gpu/drm/xe/xe_hw_engine.c b/drivers/gpu/drm/xe/xe_hw_engine.c index 87d60c4117bd..010499766fce 100644 --- a/drivers/gpu/drm/xe/xe_hw_engine.c +++ b/drivers/gpu/drm/xe/xe_hw_engine.c @@ -647,10 +647,6 @@ static int hw_engine_init(struct xe_gt *gt, struct xe_hw_engine *hwe, xe_hw_engine_enable_ring(hwe); } - /* We reserve the highest BCS instance for USM */ - if (xe->info.has_usm && hwe->class == XE_ENGINE_CLASS_COPY) - gt->usm.reserved_bcs_instance = hwe->instance; - /* Ensure IDLEDLY is lower than MAXCNT */ adjust_idledly(hwe); @@ -662,20 +658,80 @@ err_name: return err; } -static void hw_engine_setup_logical_mapping(struct xe_gt *gt) +static int hw_engine_setup_logical_and_paging_mapping(struct xe_gt *gt) { + struct xe_device *xe = gt_to_xe(gt); + unsigned int num_copy_engines = 0, num_paging_engines = 0; + unsigned int reserved_logical_bcs_start; + struct xe_hw_engine *hwe; + enum xe_hw_engine_id id; int class; + for_each_hw_engine(hwe, gt, id) + if (hwe->class == XE_ENGINE_CLASS_COPY) + num_copy_engines++; + + if (num_copy_engines && xe->info.has_usm) + num_paging_engines = 1; + + if (IS_SRIOV_VF(xe)) { + u32 vf_num_paging_engines; + + /* + * PF could in theory reserve multiple paging engines, which + * internally the submission/scheduling backend can load balance + * from. Not something we currently expect, but we are at the + * mercy of the PF, so we just need try our best to mirror the + * paging configuration. + */ + vf_num_paging_engines = xe_gt_sriov_vf_paging_engines(gt); + if (vf_num_paging_engines) { + /* This should only be non-zero on NVL-S+ */ + if (xe_gt_WARN_ON(gt, xe->info.platform < XE_NOVALAKE_S)) + return -EINVAL; + + num_paging_engines = vf_num_paging_engines; + } + } + + if (xe_gt_WARN_ON(gt, num_paging_engines > num_copy_engines)) + return -EINVAL; + + /* + * On PF, we just reserve the highest BCS instance for USM. + * + * Note: This is now a requirement going forward. The PF must ALWAYS + * reserve BCS instances in top-down order, that way the VF has a chance + * of discovering the physical BCS instance mappings for paging engines, + * in conjunction with vf_num_paging_engines. In some places we might + * only have the physical instance, and from hw pov there is no such + * thing as a paging engine. For example, the page fault descriptor, + * which comes directly from the hw, will use the physical engine + * instance. + */ + reserved_logical_bcs_start = num_copy_engines - num_paging_engines; + /* FIXME: Doing a simple logical mapping that works for most hardware */ for (class = 0; class < XE_ENGINE_CLASS_MAX; ++class) { - struct xe_hw_engine *hwe; - enum xe_hw_engine_id id; int logical_instance = 0; - for_each_hw_engine(hwe, gt, id) - if (hwe->class == class) + for_each_hw_engine(hwe, gt, id) { + if (hwe->class == class) { hwe->logical_instance = logical_instance++; + + if (class == XE_ENGINE_CLASS_COPY && + hwe->logical_instance >= + reserved_logical_bcs_start) { + if (!gt->usm.paging_hwe0) + gt->usm.paging_hwe0 = hwe; + gt->usm.paging_logical_mask |= + BIT(hwe->logical_instance); + } + } + } } + + return 0; } static void read_media_fuses(struct xe_gt *gt) @@ -894,7 +950,10 @@ int xe_hw_engines_init(struct xe_gt *gt) return err; } - hw_engine_setup_logical_mapping(gt); + err = hw_engine_setup_logical_and_paging_mapping(gt); + if (err) + return err; + err = xe_hw_engine_setup_groups(gt); if (err) return err; @@ -1043,8 +1102,7 @@ bool xe_hw_engine_is_reserved(struct xe_hw_engine *hwe) hwe->logical_instance >= gt->ccs_mode) return true; - return xe->info.has_usm && hwe->class == XE_ENGINE_CLASS_COPY && - hwe->instance == gt->usm.reserved_bcs_instance; + return xe_gt_is_usm_hwe(gt, hwe); } const char *xe_hw_engine_class_to_str(enum xe_engine_class class) diff --git a/drivers/gpu/drm/xe/xe_hw_engine_group.c b/drivers/gpu/drm/xe/xe_hw_engine_group.c index 02cf32ae5aa9..0804b426b6f9 100644 --- a/drivers/gpu/drm/xe/xe_hw_engine_group.c +++ b/drivers/gpu/drm/xe/xe_hw_engine_group.c @@ -34,6 +34,15 @@ hw_engine_group_resume_lr_jobs_func(struct work_struct *w) if (!xe_vm_in_fault_mode(q->vm)) continue; + /* + * Only resume queues that were actually suspended. A queue whose + * suspend() failed (e.g. killed/banned/wedged) was never + * suspended, so it must not be resumed. + */ + if (!READ_ONCE(q->lr.suspended)) + continue; + + WRITE_ONCE(q->lr.suspended, false); q->ops->resume(q); } @@ -140,7 +149,18 @@ int xe_hw_engine_group_add_exec_queue(struct xe_hw_engine_group *group, struct x return err; if (xe_vm_in_fault_mode(q->vm) && group->cur_mode == EXEC_MODE_DMA_FENCE) { - q->ops->suspend(q); + /* + * suspend() can fail (e.g. killed/banned/wedged), leaving the + * queue un-suspended. Propagate the failure so the queue is not + * added; on failure nothing was suspended, so there is nothing to + * undo. Only record the queue as suspended (and later resume it) + * once suspend() has succeeded. + */ + err = q->ops->suspend(q); + if (err) + goto err_suspend; + + WRITE_ONCE(q->lr.suspended, true); err = q->ops->suspend_wait(q); if (err) goto err_suspend; @@ -216,8 +236,22 @@ static int xe_hw_engine_group_suspend_faulting_lr_jobs(struct xe_hw_engine_group return -EAGAIN; xe_gt_stats_incr(q->gt, XE_GT_STATS_ID_HW_ENGINE_GROUP_SUSPEND_LR_QUEUE_COUNT, 1); + /* + * suspend() only fails when the queue is killed/banned/wedged. + * Such a queue is being torn down (its removal from HW is handled + * by the kill/ban teardown), so it is not a live fault-mode + * context the mode switch must preempt. Skip it rather than + * failing the switch, otherwise one dying sibling would block a + * dma-fence submission on the healthy queues in the group. Only + * queues recorded as suspended below are later waited on and + * resumed. + */ + err = q->ops->suspend(q); + if (err) + continue; + + WRITE_ONCE(q->lr.suspended, true); need_resume = true; - q->ops->suspend(q); gt = q->gt; } @@ -225,9 +259,13 @@ static int xe_hw_engine_group_suspend_faulting_lr_jobs(struct xe_hw_engine_group if (!xe_vm_in_fault_mode(q->vm)) continue; + /* Only wait on queues that were actually suspended above. */ + if (!READ_ONCE(q->lr.suspended)) + continue; + err = q->ops->suspend_wait(q); if (err) - return err; + goto err_resume; } if (gt) { @@ -240,6 +278,47 @@ static int xe_hw_engine_group_suspend_faulting_lr_jobs(struct xe_hw_engine_group xe_hw_engine_group_resume_faulting_lr_jobs(group); return 0; + +err_resume: + /* + * A suspend_wait() failed partway through the mode switch. Resume the + * sibling queues that were already suspended in this call so they are + * not left suspended forever. + * + * resume() requires the suspend to have completed (suspend_pending + * cleared) or it trips the !suspend_pending assert. So skip the resume + * when either: + * - suspend_wait_blocking() fails: on a GuC timeout it bans the queue + * and triggers cleanup, so the queue is being torn down; or + * - reset_status() is true: the queue was reset/killed/banned/wedged. + * suspend_wait() can return success in this case via its killed/ + * stopped wait condition while suspend_pending is still set, and the + * queue is being torn down anyway, so its state is resolved by + * teardown rather than by a resume here. + * In either case leave the queue marked suspended. + * + * Use the *blocking* (uninterruptible) wait here: the queues resumed on + * this path may belong to a different process than the one that + * triggered the mode switch. An interruptible suspend_wait() would + * return -ERESTARTSYS if the triggering task is signalled, skip the + * resume, and leave the other process's queue suspended forever + * (cross-process DoS). + */ + list_for_each_entry(q, &group->exec_queue_list, hw_engine_group_link) { + if (!xe_vm_in_fault_mode(q->vm)) + continue; + + if (!READ_ONCE(q->lr.suspended)) + continue; + + if (q->ops->suspend_wait_blocking(q) || q->ops->reset_status(q)) + continue; + + WRITE_ONCE(q->lr.suspended, false); + q->ops->resume(q); + } + + return err; } /** diff --git a/drivers/gpu/drm/xe/xe_hw_engine_types.h b/drivers/gpu/drm/xe/xe_hw_engine_types.h index 84c097da9b6f..ff115ab429fb 100644 --- a/drivers/gpu/drm/xe/xe_hw_engine_types.h +++ b/drivers/gpu/drm/xe/xe_hw_engine_types.h @@ -114,7 +114,12 @@ struct xe_hw_engine { enum xe_engine_class class; /** @instance: physical instance of this hw engine */ u16 instance; - /** @logical_instance: logical instance of this hw engine */ + /** + * @logical_instance: logical instance of this hw engine. + * + * Note: For GuC usage, always use xe_hwe_guc_logical_instance(). + * For GuC usage, we should no longer use the raw logical instance. + */ u16 logical_instance; /** @irq_offset: IRQ offset of this hw engine */ u16 irq_offset; diff --git a/drivers/gpu/drm/xe/xe_hw_error.c b/drivers/gpu/drm/xe/xe_hw_error.c index 4a4b363fc844..5f2abc9485ff 100644 --- a/drivers/gpu/drm/xe/xe_hw_error.c +++ b/drivers/gpu/drm/xe/xe_hw_error.c @@ -4,12 +4,12 @@ */ #include <linux/bitmap.h> -#include <linux/fault-inject.h> #include "regs/xe_gsc_regs.h" #include "regs/xe_hw_error_regs.h" #include "regs/xe_irq_regs.h" +#include "xe_debugfs.h" #include "xe_device.h" #include "xe_drm_ras.h" #include "xe_hw_error.h" @@ -25,8 +25,6 @@ (PVC_COR_ERR_MASK & REG_BIT(err_bit)) : \ (PVC_FAT_ERR_MASK & REG_BIT(err_bit))) -extern struct fault_attr inject_csc_hw_error; - static const char * const error_severity[] = DRM_XE_RAS_ERROR_SEVERITY_NAMES; static const char * const hec_uncorrected_fw_errors[] = { @@ -167,11 +165,6 @@ static_assert(ARRAY_SIZE(pvc_master_local_nonfatal_err_reg) == XE_RAS_REG_SIZE); pvc_master_local_fatal_err_reg : \ pvc_master_local_nonfatal_err_reg) -static bool fault_inject_csc_hw_error(void) -{ - return IS_ENABLED(CONFIG_DEBUG_FS) && should_fail(&inject_csc_hw_error, 1); -} - static void csc_hw_error_work(struct work_struct *work) { struct xe_tile *tile = container_of(work, typeof(*tile), csc_hw_error_work); @@ -517,7 +510,7 @@ void xe_hw_error_irq_handler(struct xe_tile *tile, const u32 master_ctl) { enum hardware_error hw_err; - if (fault_inject_csc_hw_error()) + if (xe_fault_csc_hw_error()) schedule_work(&tile->csc_hw_error_work); for (hw_err = 0; hw_err < HARDWARE_ERROR_MAX; hw_err++) { diff --git a/drivers/gpu/drm/xe/xe_i2c.c b/drivers/gpu/drm/xe/xe_i2c.c index bd956776b10b..a26c38bb17a1 100644 --- a/drivers/gpu/drm/xe/xe_i2c.c +++ b/drivers/gpu/drm/xe/xe_i2c.c @@ -95,18 +95,21 @@ static int xe_i2c_register_adapter(struct xe_i2c *i2c) struct platform_device *pdev; struct fwnode_handle *fwnode; int ret; + u32 id; fwnode = fwnode_create_software_node(xe_i2c_adapter_properties, NULL); if (IS_ERR(fwnode)) return PTR_ERR(fwnode); + id = (pci_domain_nr(pci->bus) << 16) | pci_dev_id(pci); + /* * Not using platform_device_register_full() here because we don't have * a handle to the platform_device before it returns. xe_i2c_notifier() * uses that handle, but it may be called before * platform_device_register_full() is done. */ - pdev = platform_device_alloc(adapter_name, pci_dev_id(pci)); + pdev = platform_device_alloc(adapter_name, id); if (!pdev) { ret = -ENOMEM; goto err_fwnode_remove; diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c index 9428dd5e7760..f79d0047bec6 100644 --- a/drivers/gpu/drm/xe/xe_migrate.c +++ b/drivers/gpu/drm/xe/xe_migrate.c @@ -117,6 +117,27 @@ static void xe_migrate_fini(void *arg) xe_exec_queue_put(m->q); } +static inline u16 xe_migrate_pat_index(struct xe_device *xe, + enum ttm_caching caching, + bool is_comp_pte) +{ + enum xe_cache_level cache_level; + + /* + * Select the appropriate PAT index for buffer object PTEs programmed + * by emit_pte(). We choose not to mess with xe_migrate_prepare_vm() + * yet, for simplicity. + */ + if (is_comp_pte && GRAPHICS_VERx100(xe) >= 2000) + cache_level = XE_CACHE_NONE_COMPRESSION; + else if (caching == ttm_cached) + cache_level = XE_CACHE_WB; + else + cache_level = XE_CACHE_NONE; + + return xe_cache_pat_idx(xe, cache_level); +} + static u64 xe_migrate_vm_addr(u64 slot, u32 level) { XE_WARN_ON(slot >= NUM_PT_SLOTS); @@ -383,27 +404,6 @@ static void xe_migrate_suballoc_manager_init(struct xe_migrate *m, u32 map_ofs) NUM_VMUSA_UNIT_PER_PAGE, 0); } -/* - * Including the reserved copy engine is required to avoid deadlocks due to - * migrate jobs servicing the faults gets stuck behind the job that faulted. - */ -static u32 xe_migrate_usm_logical_mask(struct xe_gt *gt) -{ - u32 logical_mask = 0; - struct xe_hw_engine *hwe; - enum xe_hw_engine_id id; - - for_each_hw_engine(hwe, gt, id) { - if (hwe->class != XE_ENGINE_CLASS_COPY) - continue; - - if (xe_gt_is_usm_hwe(gt, hwe)) - logical_mask |= BIT(hwe->logical_instance); - } - - return logical_mask; -} - static bool xe_migrate_needs_ccs_emit(struct xe_device *xe) { return xe_device_has_flat_ccs(xe) && !(GRAPHICS_VER(xe) >= 20 && IS_DGFX(xe)); @@ -479,13 +479,10 @@ int xe_migrate_init(struct xe_migrate *m) goto err_out; if (xe->info.has_usm) { - struct xe_hw_engine *hwe = xe_gt_hw_engine(primary_gt, - XE_ENGINE_CLASS_COPY, - primary_gt->usm.reserved_bcs_instance, - false); - u32 logical_mask = xe_migrate_usm_logical_mask(primary_gt); + struct xe_hw_engine *hwe0 = primary_gt->usm.paging_hwe0; + u32 logical_mask = primary_gt->usm.paging_logical_mask; - if (!hwe || !logical_mask) { + if (!hwe0 || !logical_mask) { err = -EINVAL; goto err_out; } @@ -494,7 +491,7 @@ int xe_migrate_init(struct xe_migrate *m) * XXX: Currently only reserving 1 (likely slow) BCS instance on * PVC, may want to revisit if performance is needed. */ - m->q = xe_exec_queue_create(xe, vm, logical_mask, 1, hwe, + m->q = xe_exec_queue_create(xe, vm, logical_mask, 1, hwe0, EXEC_QUEUE_FLAG_KERNEL | EXEC_QUEUE_FLAG_PERMANENT | EXEC_QUEUE_FLAG_HIGH_PRIORITY | @@ -631,17 +628,17 @@ static void emit_pte(struct xe_migrate *m, { struct xe_device *xe = tile_to_xe(m->tile); struct xe_vm *vm = m->q->vm; + struct xe_bo *bo = ttm_to_xe_bo(res->bo); + enum ttm_caching caching = ttm_cached; u16 pat_index; u32 ptes; u64 ofs = (u64)at_pt * XE_PAGE_SIZE; u64 cur_ofs; - /* Indirect access needs compression enabled uncached PAT index */ - if (GRAPHICS_VERx100(xe) >= 2000) - pat_index = is_comp_pte ? xe_cache_pat_idx(xe, XE_CACHE_NONE_COMPRESSION) : - xe_cache_pat_idx(xe, XE_CACHE_WB); - else - pat_index = xe_cache_pat_idx(xe, XE_CACHE_WB); + if (!is_vram && bo->ttm.ttm) + caching = bo->ttm.ttm->caching; + + pat_index = xe_migrate_pat_index(xe, caching, is_comp_pte); ptes = DIV_ROUND_UP(size, XE_PAGE_SIZE); @@ -1166,6 +1163,8 @@ static int emit_flush_invalidate(u32 *dw, int i, u32 flags) * @tile: Tile whose migration context to be used. * @q : Execution to be used along with migration context. * @src_bo: The buffer object @src is currently bound to. + * @new_mem: The (not yet committed) destination resource @src_bo is being + * moved into; src_bo->ttm.resource is still the old resource. * @read_write : Creates BB commands for CCS read/write. * * Creates batch buffer instructions to copy CCS metadata from CCS pool to @@ -1177,12 +1176,13 @@ static int emit_flush_invalidate(u32 *dw, int i, u32 flags) */ int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q, struct xe_bo *src_bo, + struct ttm_resource *new_mem, enum xe_sriov_vf_ccs_rw_ctxs read_write) { bool src_is_pltt = read_write == XE_SRIOV_VF_CCS_READ_CTX; bool dst_is_pltt = read_write == XE_SRIOV_VF_CCS_WRITE_CTX; - struct ttm_resource *src = src_bo->ttm.resource; + struct ttm_resource *src = new_mem; struct xe_migrate *m = tile->migrate; struct xe_gt *gt = tile->primary_gt; u32 batch_size, batch_size_allocated; @@ -1310,6 +1310,7 @@ int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q, * content. * @src_bo: The buffer object @src is currently bound to. * @read_write : Creates BB commands for CCS read/write. + * @bound: Device is bound * * Directly clearing the BB lacks atomicity and can lead to undefined * behavior if the vCPU is halted mid-operation during the clearing @@ -1322,7 +1323,8 @@ int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q, * Returns: None. */ void xe_migrate_ccs_rw_copy_clear(struct xe_bo *src_bo, - enum xe_sriov_vf_ccs_rw_ctxs read_write) + enum xe_sriov_vf_ccs_rw_ctxs read_write, + bool bound) { struct xe_mem_pool_node *bb = src_bo->bb_ccs[read_write]; struct xe_device *xe = xe_bo_device(src_bo); @@ -1336,13 +1338,15 @@ void xe_migrate_ccs_rw_copy_clear(struct xe_bo *src_bo, bb_pool = ctx->mem.ccs_bb_pool; scoped_guard(mutex, xe_mem_pool_bo_swap_guard(bb_pool)) { - xe_mem_pool_swap_shadow_locked(bb_pool); + if (bound) { + xe_mem_pool_swap_shadow_locked(bb_pool); - cs = xe_mem_pool_node_cpu_addr(bb); - memset(cs, MI_NOOP, bb->sa_node.size); - xe_sriov_vf_ccs_rw_update_bb_addr(ctx); + cs = xe_mem_pool_node_cpu_addr(bb); + memset(cs, MI_NOOP, bb->sa_node.size); + xe_sriov_vf_ccs_rw_update_bb_addr(ctx); - xe_mem_pool_sync_shadow_locked(bb); + xe_mem_pool_sync_shadow_locked(bb); + } xe_mem_pool_free_node(bb); src_bo->bb_ccs[read_write] = NULL; } diff --git a/drivers/gpu/drm/xe/xe_migrate.h b/drivers/gpu/drm/xe/xe_migrate.h index 965c45889c72..c3a268b01768 100644 --- a/drivers/gpu/drm/xe/xe_migrate.h +++ b/drivers/gpu/drm/xe/xe_migrate.h @@ -138,10 +138,12 @@ struct dma_fence *xe_migrate_resolve(struct xe_migrate *m, int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q, struct xe_bo *src_bo, + struct ttm_resource *new_mem, enum xe_sriov_vf_ccs_rw_ctxs read_write); void xe_migrate_ccs_rw_copy_clear(struct xe_bo *src_bo, - enum xe_sriov_vf_ccs_rw_ctxs read_write); + enum xe_sriov_vf_ccs_rw_ctxs read_write, + bool bound); struct xe_lrc *xe_migrate_lrc(struct xe_migrate *migrate); struct xe_exec_queue *xe_migrate_exec_queue(struct xe_migrate *migrate); diff --git a/drivers/gpu/drm/xe/xe_module.c b/drivers/gpu/drm/xe/xe_module.c index 39e4fc85f019..848d65265443 100644 --- a/drivers/gpu/drm/xe/xe_module.c +++ b/drivers/gpu/drm/xe/xe_module.c @@ -7,6 +7,7 @@ #include <linux/init.h> #include <linux/module.h> +#include <linux/workqueue.h> #include <drm/drm_module.h> @@ -88,6 +89,50 @@ static int xe_check_nomodeset(void) return 0; } +static struct workqueue_struct *xe_destroy_wq; + +static int __init xe_destroy_wq_module_init(void) +{ + xe_destroy_wq = alloc_workqueue("xe-guc-destroy-wq", WQ_UNBOUND, 0); + if (!xe_destroy_wq) + return -ENOMEM; + return 0; +} + +static void xe_destroy_wq_module_exit(void) +{ + if (xe_destroy_wq) + destroy_workqueue(xe_destroy_wq); + xe_destroy_wq = NULL; +} + +/** + * xe_destroy_wq_queue() - Queue work on the destroy workqueue + * @work: work item to queue + * + * The destroy workqueue has module lifetime and is used for GuC exec queue + * teardown that can outlive a single xe_device. SVM pagemap destroy uses the + * per-device xe->destroy_wq instead. + * + * Return: %true if @work was queued, %false if it was already pending. + */ +bool xe_destroy_wq_queue(struct work_struct *work) +{ + return queue_work(xe_destroy_wq, work); +} + +/** + * xe_destroy_wq_flush() - Flush the destroy workqueue + * + * Drains all pending destroy work. Called from PCI remove to ensure + * teardown ordering before the device is destroyed. + */ +void xe_destroy_wq_flush(void) +{ + if (xe_destroy_wq) + flush_workqueue(xe_destroy_wq); +} + struct init_funcs { int (*init)(void); void (*exit)(void); @@ -110,6 +155,10 @@ static const struct init_funcs init_funcs[] = { .exit = xe_sched_job_module_exit, }, { + .init = xe_destroy_wq_module_init, + .exit = xe_destroy_wq_module_exit, + }, + { .init = xe_register_pci_driver, .exit = xe_unregister_pci_driver, }, diff --git a/drivers/gpu/drm/xe/xe_module.h b/drivers/gpu/drm/xe/xe_module.h index c75153471248..a0eb7db07770 100644 --- a/drivers/gpu/drm/xe/xe_module.h +++ b/drivers/gpu/drm/xe/xe_module.h @@ -8,6 +8,8 @@ #include <linux/types.h> +struct work_struct; + /* Module modprobe variables */ struct xe_modparam { bool probe_display; @@ -26,5 +28,8 @@ struct xe_modparam { extern struct xe_modparam xe_modparam; +bool xe_destroy_wq_queue(struct work_struct *work); +void xe_destroy_wq_flush(void); + #endif diff --git a/drivers/gpu/drm/xe/xe_nvm.c b/drivers/gpu/drm/xe/xe_nvm.c index 33487e91f366..1ea67eaeae24 100644 --- a/drivers/gpu/drm/xe/xe_nvm.c +++ b/drivers/gpu/drm/xe/xe_nvm.c @@ -60,35 +60,40 @@ static bool xe_nvm_writable_override(struct xe_device *xe) struct xe_mmio *mmio = xe_root_tile_mmio(xe); bool writable_override; struct xe_reg reg; - u32 test_bit; + u32 test_bit, test_val; switch (xe->info.platform) { case XE_CRESCENTISLAND: reg = PCODE_SCRATCH(0); test_bit = FDO_MODE; + test_val = FDO_MODE; break; case XE_BATTLEMAGE: reg = HECI_FWSTS2(DG2_GSC_HECI2_BASE); test_bit = HECI_FW_STATUS_2_NVM_ACCESS_MODE; + test_val = 0; break; case XE_PVC: reg = HECI_FWSTS2(PVC_GSC_HECI2_BASE); test_bit = HECI_FW_STATUS_2_NVM_ACCESS_MODE; + test_val = 0; break; case XE_DG2: reg = HECI_FWSTS2(DG2_GSC_HECI2_BASE); test_bit = HECI_FW_STATUS_2_NVM_ACCESS_MODE; + test_val = 0; break; case XE_DG1: reg = HECI_FWSTS2(DG1_GSC_HECI2_BASE); test_bit = HECI_FW_STATUS_2_NVM_ACCESS_MODE; + test_val = 0; break; default: drm_err(&xe->drm, "Unknown platform\n"); return true; } - writable_override = !(xe_mmio_read32(mmio, reg) & test_bit); + writable_override = (xe_mmio_read32(mmio, reg) & test_bit) == test_val; if (writable_override) drm_info(&xe->drm, "NVM access overridden by jumper\n"); return writable_override; diff --git a/drivers/gpu/drm/xe/xe_pat.c b/drivers/gpu/drm/xe/xe_pat.c index fad5b5a5ed4a..a5fe1beec652 100644 --- a/drivers/gpu/drm/xe/xe_pat.c +++ b/drivers/gpu/drm/xe/xe_pat.c @@ -25,6 +25,7 @@ 0x4800, 0x4804, \ 0x4848, 0x484c) #define _PAT_PTA 0x4820 +#define _PAT_TR_PTA 0x48cc #define XE2_NO_PROMOTE REG_BIT(10) #define XE2_COMP_EN REG_BIT(9) @@ -256,6 +257,7 @@ static const struct xe_pat_table_entry xe3p_xpc_pat_table[] = { static const struct xe_pat_table_entry xe3p_primary_pat_pta = XE2_PAT(0, 0, 0, 0, 0, 3); static const struct xe_pat_table_entry xe3p_media_pat_pta = XE2_PAT(0, 0, 0, 0, 0, 2); +static const struct xe_pat_table_entry xe3p_pat_tr_pta = XE2_PAT(0, 0, 0, 0, 0, 0); static const struct xe_pat_table_entry xe3p_lpg_pat_table[] = { [ 0] = XE2_PAT( 0, 0, 0, 0, 3, 0 ), @@ -325,11 +327,26 @@ static const struct xe_pat_table_entry *gt_pta_entry(struct xe_gt *gt) return NULL; } +static const struct xe_pat_table_entry *gt_tr_pta_entry(struct xe_gt *gt) +{ + struct xe_device *xe = gt_to_xe(gt); + + if (xe_gt_is_main_type(gt)) + return xe->pat.pat_primary_tr_pta; + + if (xe_gt_is_media_type(gt)) + return xe->pat.pat_media_tr_pta; + + xe_assert(xe, false); + return NULL; +} + static void program_pat(struct xe_gt *gt, const struct xe_pat_table_entry table[], int n_entries) { struct xe_device *xe = gt_to_xe(gt); const struct xe_pat_table_entry *pta_entry = gt_pta_entry(gt); + const struct xe_pat_table_entry *tr_pta_entry = gt_tr_pta_entry(gt); for (int i = 0; i < n_entries; i++) { struct xe_reg reg = XE_REG(_PAT_INDEX(i)); @@ -342,6 +359,9 @@ static void program_pat(struct xe_gt *gt, const struct xe_pat_table_entry table[ if (pta_entry) xe_mmio_write32(>->mmio, XE_REG(_PAT_PTA), pta_entry->value); + + if (tr_pta_entry) + xe_mmio_write32(>->mmio, XE_REG(_PAT_TR_PTA), tr_pta_entry->value); } static void program_pat_mcr(struct xe_gt *gt, const struct xe_pat_table_entry table[], @@ -349,6 +369,7 @@ static void program_pat_mcr(struct xe_gt *gt, const struct xe_pat_table_entry ta { struct xe_device *xe = gt_to_xe(gt); const struct xe_pat_table_entry *pta_entry = gt_pta_entry(gt); + const struct xe_pat_table_entry *tr_pta_entry = gt_tr_pta_entry(gt); for (int i = 0; i < n_entries; i++) { struct xe_reg_mcr reg_mcr = XE_REG_MCR(_PAT_INDEX(i)); @@ -361,6 +382,9 @@ static void program_pat_mcr(struct xe_gt *gt, const struct xe_pat_table_entry ta if (pta_entry) xe_gt_mcr_multicast_write(gt, XE_REG_MCR(_PAT_PTA), pta_entry->value); + + if (tr_pta_entry) + xe_gt_mcr_multicast_write(gt, XE_REG_MCR(_PAT_TR_PTA), tr_pta_entry->value); } static int xelp_dump(struct xe_gt *gt, struct drm_printer *p) @@ -531,6 +555,16 @@ static int xe2_dump(struct xe_gt *gt, struct drm_printer *p) drm_printf(p, "Page Table Access:\n"); xe->pat.ops->entry_dump(p, "PTA_MODE", pat, false); + if (gt_tr_pta_entry(gt)) { + if (xe_gt_is_media_type(gt)) + pat = xe_mmio_read32(>->mmio, XE_REG(_PAT_TR_PTA)); + else + pat = xe_gt_mcr_unicast_read_any(gt, XE_REG_MCR(_PAT_TR_PTA)); + + drm_printf(p, "TRTT Page Table Access:\n"); + xe->pat.ops->entry_dump(p, "TR_PTA_MODE", pat, false); + } + if (xe_gt_is_media_type(gt)) pat = xe_mmio_read32(>->mmio, XE_REG(_PAT_ATS)); else @@ -577,6 +611,8 @@ void xe_pat_init_early(struct xe_device *xe) if (!IS_DGFX(xe)) { xe->pat.pat_primary_pta = &xe3p_primary_pat_pta; xe->pat.pat_media_pta = &xe3p_media_pat_pta; + xe->pat.pat_primary_tr_pta = &xe3p_pat_tr_pta; + xe->pat.pat_media_tr_pta = &xe3p_pat_tr_pta; } xe->pat.n_entries = ARRAY_SIZE(xe3p_lpg_pat_table); xe->pat.idx[XE_CACHE_NONE] = 3; @@ -701,6 +737,7 @@ int xe_pat_dump_sw_config(struct xe_gt *gt, struct drm_printer *p) { struct xe_device *xe = gt_to_xe(gt); const struct xe_pat_table_entry *pta_entry = gt_pta_entry(gt); + const struct xe_pat_table_entry *tr_pta_entry = gt_tr_pta_entry(gt); char label[PAT_LABEL_LEN]; if (!xe->pat.table || !xe->pat.n_entries) @@ -731,6 +768,13 @@ int xe_pat_dump_sw_config(struct xe_gt *gt, struct drm_printer *p) xe->pat.ops->entry_dump(p, "PTA_MODE", pat, false); } + if (tr_pta_entry) { + u32 pat = tr_pta_entry->value; + + drm_printf(p, "TRTT Page Table Access:\n"); + xe->pat.ops->entry_dump(p, "TR_PTA_MODE", pat, false); + } + if (xe->pat.pat_ats) { u32 pat = xe->pat.pat_ats->value; diff --git a/drivers/gpu/drm/xe/xe_pci.c b/drivers/gpu/drm/xe/xe_pci.c index 91af603e9431..5f2a0b19839d 100644 --- a/drivers/gpu/drm/xe/xe_pci.c +++ b/drivers/gpu/drm/xe/xe_pci.c @@ -22,6 +22,7 @@ #include "xe_device.h" #include "xe_drv.h" #include "xe_gt.h" +#include "xe_gt_printk.h" #include "xe_gt_sriov_vf.h" #include "xe_guc.h" #include "xe_mmio.h" @@ -36,6 +37,7 @@ #include "xe_step.h" #include "xe_survivability_mode.h" #include "xe_tile.h" +#include "xe_tile_printk.h" enum toggle_d3cold { D3COLD_DISABLE, @@ -120,6 +122,7 @@ static const struct xe_graphics_desc graphics_xe2 = { static const struct xe_graphics_desc graphics_xe3p_lpg = { XE2_GFX_FEATURES, .has_indirect_ring_state = 1, + .has_uncorrectable_error_reporting = 1, .multi_queue_engine_class_mask = BIT(XE_ENGINE_CLASS_COPY) | BIT(XE_ENGINE_CLASS_COMPUTE), .num_geometry_xecore_fuse_regs = 3, .num_compute_xecore_fuse_regs = 3, @@ -129,6 +132,7 @@ static const struct xe_graphics_desc graphics_xe3p_xpc = { XE2_GFX_FEATURES, .has_access_counter = 0, .has_indirect_ring_state = 1, + .has_uncorrectable_error_reporting = 1, .hw_engine_mask = GENMASK(XE_HW_ENGINE_BCS8, XE_HW_ENGINE_BCS1) | GENMASK(XE_HW_ENGINE_CCS3, XE_HW_ENGINE_CCS0), @@ -151,6 +155,14 @@ static const struct xe_media_desc media_xelpmp = { BIT(XE_HW_ENGINE_GSCCS0) }; +static const struct xe_media_desc media_xe3p_hpm = { + .has_uncorrectable_error_reporting = 1, + .hw_engine_mask = + GENMASK(XE_HW_ENGINE_VCS7, XE_HW_ENGINE_VCS0) | + GENMASK(XE_HW_ENGINE_VECS3, XE_HW_ENGINE_VECS0) | + BIT(XE_HW_ENGINE_GSCCS0) +}; + /* Pre-GMDID Graphics IPs */ static const struct xe_ip graphics_ip_xelp = { 1200, "Xe_LP", &graphics_xelp }; static const struct xe_ip graphics_ip_xelpp = { 1210, "Xe_LP+", &graphics_xelp }; @@ -186,7 +198,7 @@ static const struct xe_ip media_ips[] = { { 3000, "Xe3_LPM", &media_xelpmp }, { 3002, "Xe3_LPM", &media_xelpmp }, { 3500, "Xe3p_LPM", &media_xelpmp }, - { 3503, "Xe3p_HPM", &media_xelpmp }, + { 3503, "Xe3p_HPM", &media_xe3p_hpm }, }; #define MULTI_LRC_MASK \ @@ -449,7 +461,6 @@ static const struct xe_device_desc nvls_desc = { .has_sriov = true, .max_gt_per_tile = 2, MULTI_LRC_MASK, - .require_force_probe = true, .va_bits = 48, .vm_max_level = 4, }; @@ -809,7 +820,8 @@ static int xe_info_init_early(struct xe_device *xe, xe->info.probe_display = IS_ENABLED(CONFIG_DRM_XE_DISPLAY) && xe_modparam.probe_display && - desc->has_display; + desc->has_display && + !xe_device_is_admin_only(xe); xe_assert(xe, desc->max_gt_per_tile > 0); xe_assert(xe, desc->max_gt_per_tile <= XE_MAX_GT_PER_TILE); @@ -864,7 +876,7 @@ static struct xe_gt *alloc_primary_gt(struct xe_tile *tile, struct xe_gt *gt; if (!xe_configfs_primary_gt_allowed(to_pci_dev(xe->drm.dev))) { - xe_info(xe, "Primary GT disabled via configfs\n"); + xe_tile_info(tile, "Primary GT disabled via configfs\n"); return NULL; } @@ -875,7 +887,13 @@ static struct xe_gt *alloc_primary_gt(struct xe_tile *tile, gt->info.type = XE_GT_TYPE_MAIN; gt->info.id = tile->id * xe->info.max_gt_per_tile; gt->info.has_indirect_ring_state = graphics_desc->has_indirect_ring_state; + gt->info.has_uncorrectable_error_reporting = + graphics_desc->has_uncorrectable_error_reporting; gt->info.multi_queue_engine_class_mask = graphics_desc->multi_queue_engine_class_mask; + if (!xe_configfs_get_enable_multi_queue(to_pci_dev(xe->drm.dev))) { + xe_gt_info(gt, "Multi-queue disabled via configfs\n"); + gt->info.multi_queue_engine_class_mask = 0; + } gt->info.engine_mask = graphics_desc->hw_engine_mask; gt->info.num_geometry_xecore_fuse_regs = graphics_desc->num_geometry_xecore_fuse_regs; gt->info.num_compute_xecore_fuse_regs = graphics_desc->num_compute_xecore_fuse_regs; @@ -906,7 +924,7 @@ static struct xe_gt *alloc_media_gt(struct xe_tile *tile, struct xe_gt *gt; if (!xe_configfs_media_gt_allowed(to_pci_dev(xe->drm.dev))) { - xe_info(xe, "Media GT disabled via configfs\n"); + xe_tile_info(tile, "Media GT disabled via configfs\n"); return NULL; } @@ -920,6 +938,7 @@ static struct xe_gt *alloc_media_gt(struct xe_tile *tile, gt->info.type = XE_GT_TYPE_MEDIA; gt->info.id = tile->id * xe->info.max_gt_per_tile + 1; gt->info.has_indirect_ring_state = media_desc->has_indirect_ring_state; + gt->info.has_uncorrectable_error_reporting = media_desc->has_uncorrectable_error_reporting; gt->info.engine_mask = media_desc->hw_engine_mask; return gt; @@ -1098,6 +1117,12 @@ static void xe_pci_remove(struct pci_dev *pdev) return; xe_device_remove(xe); + + /* + * Preserve remove-time flush after moving destroy work to module + * lifetime. + */ + xe_destroy_wq_flush(); xe_pm_fini(xe); } diff --git a/drivers/gpu/drm/xe/xe_pci_error.c b/drivers/gpu/drm/xe/xe_pci_error.c index 9b78cc0d3293..e41af2ac7f23 100644 --- a/drivers/gpu/drm/xe/xe_pci_error.c +++ b/drivers/gpu/drm/xe/xe_pci_error.c @@ -10,6 +10,7 @@ #include "xe_pci.h" #include "xe_pm.h" #include "xe_printk.h" +#include "xe_ras.h" #include "xe_survivability_mode.h" static void prepare_device_for_reset(struct pci_dev *pdev) @@ -34,6 +35,21 @@ static void prepare_device_for_reset(struct pci_dev *pdev) pci_disable_device(pdev); } +static pci_ers_result_t ras_action_to_pci_result(struct pci_dev *pdev, u8 action) +{ + switch (action) { + case XE_RAS_RECOVERY_ACTION_RECOVERED: + return PCI_ERS_RESULT_RECOVERED; + case XE_RAS_RECOVERY_ACTION_RESET: + prepare_device_for_reset(pdev); + return PCI_ERS_RESULT_NEED_RESET; + case XE_RAS_RECOVERY_ACTION_DISCONNECT: + return PCI_ERS_RESULT_DISCONNECT; + default: + return PCI_ERS_RESULT_DISCONNECT; + } +} + static pci_ers_result_t xe_pci_error_detected(struct pci_dev *pdev, pci_channel_state_t state) { struct xe_device *xe = pdev_to_xe_device(pdev); @@ -62,11 +78,12 @@ static pci_ers_result_t xe_pci_error_detected(struct pci_dev *pdev, pci_channel_ static pci_ers_result_t xe_pci_error_mmio_enabled(struct pci_dev *pdev) { struct xe_device *xe = pdev_to_xe_device(pdev); + enum xe_ras_recovery_action action; xe_info(xe, "PCI error: MMIO enabled\n"); + action = xe_ras_process_errors(xe); - /* TODO: Query system controller for the type of error and take appropriate action */ - return PCI_ERS_RESULT_RECOVERED; + return ras_action_to_pci_result(pdev, action); } static pci_ers_result_t xe_pci_error_slot_reset(struct pci_dev *pdev) diff --git a/drivers/gpu/drm/xe/xe_pci_types.h b/drivers/gpu/drm/xe/xe_pci_types.h index 24d4a3d00517..fed509ff601e 100644 --- a/drivers/gpu/drm/xe/xe_pci_types.h +++ b/drivers/gpu/drm/xe/xe_pci_types.h @@ -79,12 +79,14 @@ struct xe_graphics_desc { u8 has_ctx_tlb_inval:1; u8 has_usm:1; u8 has_64bit_timestamp:1; + u8 has_uncorrectable_error_reporting:1; }; struct xe_media_desc { u64 hw_engine_mask; /* hardware engines provided by media IP */ u8 has_indirect_ring_state:1; + u8 has_uncorrectable_error_reporting:1; }; struct xe_ip { diff --git a/drivers/gpu/drm/xe/xe_pcode.c b/drivers/gpu/drm/xe/xe_pcode.c index 866986694d9c..ccc3bdeed6bb 100644 --- a/drivers/gpu/drm/xe/xe_pcode.c +++ b/drivers/gpu/drm/xe/xe_pcode.c @@ -11,10 +11,14 @@ #include <drm/drm_managed.h> +#include "regs/xe_pmt.h" #include "xe_assert.h" #include "xe_device.h" #include "xe_mmio.h" #include "xe_pcode_api.h" +#include "xe_pm.h" +#include "xe_printk.h" +#include "xe_vsec.h" /** * DOC: PCODE @@ -350,3 +354,31 @@ int xe_pcode_probe_early(struct xe_device *xe) return xe_pcode_ready(xe, false); } ALLOW_ERROR_INJECTION(xe_pcode_probe_early, ERRNO); /* See xe_pci_probe */ + +/** + * xe_get_pcode_version - Read pcode version via PMT telemetry + * @xe: xe instance + * @version: pointer to struct xe_pcode_version to store version info + * + * Reads the pcode version from PMT telemetry and fills the + * provided @version structure. + * + * Return: 0 on success, negative error code on failure. + */ +int xe_get_pcode_version(struct xe_device *xe, struct xe_pcode_version *version) +{ + int ret = 0; + + guard(xe_pm_runtime)(xe); + + ret = xe_pmt_telem_read(xe->drm.dev, + xe_mmio_read32(xe_root_tile_mmio(xe), PUNIT_TELEMETRY_GUID), + (u64 *)version, PUNIT_VERSION_OFFSET, sizeof(*version)); + if (ret != sizeof(*version)) { + xe_warn(xe, "pcode version read from PMT failed, ret %pe\n", ERR_PTR(ret)); + return ret; + } + xe_dbg(xe, "pcode version major %u minor %u engg %u\n", version->major, + version->minor, version->engg); + return 0; +} diff --git a/drivers/gpu/drm/xe/xe_pcode.h b/drivers/gpu/drm/xe/xe_pcode.h index 18260c29e620..8fb3e4ba13a6 100644 --- a/drivers/gpu/drm/xe/xe_pcode.h +++ b/drivers/gpu/drm/xe/xe_pcode.h @@ -12,6 +12,12 @@ struct drm_device; struct xe_device; struct xe_tile; +struct xe_pcode_version { + u16 minor; + u16 major; + u32 engg; +}; + int xe_pcode_init_early(struct xe_tile *tile); int xe_pcode_probe_early(struct xe_device *xe); int xe_pcode_ready(struct xe_device *xe, bool locked); @@ -22,6 +28,7 @@ int xe_pcode_write_timeout(struct xe_tile *tile, u32 mbox, u32 val, int timeout_ms); int xe_pcode_write64_timeout(struct xe_tile *tile, u32 mbox, u32 data0, u32 data1, int timeout); +int xe_get_pcode_version(struct xe_device *xe, struct xe_pcode_version *version); #define xe_pcode_write(tile, mbox, val) \ xe_pcode_write_timeout(tile, mbox, val, 1) diff --git a/drivers/gpu/drm/xe/xe_preempt_fence.c b/drivers/gpu/drm/xe/xe_preempt_fence.c index d6427b473ddd..4aa570fe745d 100644 --- a/drivers/gpu/drm/xe/xe_preempt_fence.c +++ b/drivers/gpu/drm/xe/xe_preempt_fence.c @@ -74,6 +74,13 @@ static bool preempt_fence_enable_signaling(struct dma_fence *fence) struct xe_exec_queue *q = pfence->q; pfence->error = q->ops->suspend(q); + /* + * Record a successful suspend so the rebind worker only resumes queues + * that were actually suspended; a failed suspend() leaves the queue + * un-suspended and must not be paired with a resume(). + */ + if (!pfence->error) + WRITE_ONCE(q->lr.suspended, true); queue_work(q->vm->xe->preempt_fence_wq, &pfence->preempt_work); return true; } diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c index c48ddc888cb0..a07316a45d79 100644 --- a/drivers/gpu/drm/xe/xe_pt.c +++ b/drivers/gpu/drm/xe/xe_pt.c @@ -303,6 +303,14 @@ struct xe_pt_stage_bind_walk { /** @clear_pt: clear page table entries during the bind walk */ bool clear_pt; /** + * @target_leaf_level: Page-table level at which to emit leaf PTEs + * 0 for normal 4K/64K mappings, 1 for 2M huge pages, and 2 for 1G huge + * pages. The walk still traverses from the root down; this field tells + * xe_pt_stage_bind_entry() to treat the selected level as a leaf instead + * of descending further. + */ + u32 target_leaf_level; + /** * @vma: VMA being mapped */ struct xe_vma *vma; @@ -443,10 +451,6 @@ static bool xe_pt_hugepte_possible(u64 addr, u64 next, unsigned int level, if (!xe_pt_covers(addr, next, level, &xe_walk->base)) return false; - /* Does the DMA segment cover the whole pte? */ - if (next - xe_walk->va_curs_start > xe_walk->curs->size) - return false; - /* null VMA's and purged BO's do not have dma addresses */ if (xe_vma_is_null(xe_walk->vma) || (bo && xe_bo_is_purged(bo))) return true; @@ -455,6 +459,10 @@ static bool xe_pt_hugepte_possible(u64 addr, u64 next, unsigned int level, if (xe_walk->clear_pt) return true; + /* Does the DMA segment cover the whole pte? */ + if (next - xe_walk->va_curs_start > xe_walk->curs->size) + return false; + /* Is the DMA address huge PTE size aligned? */ size = next - addr; dma = addr - xe_walk->va_curs_start + xe_res_dma(xe_walk->curs); @@ -514,6 +522,39 @@ xe_pt_is_pte_ps64K(u64 addr, u64 next, struct xe_pt_stage_bind_walk *xe_walk) return xe_walk->found_64K; } +static bool xe_pt_huge_leaf_allowed(u64 addr, u64 next, unsigned int level, + struct xe_pt_stage_bind_walk *xe_walk) +{ + if (xe_walk->clear_pt) + return xe_pt_hugepte_possible(addr, next, level, xe_walk); + + if (!xe_debug_page_size_supported(xe_walk->vm->xe)) + return xe_pt_hugepte_possible(addr, next, level, xe_walk); + + if (!xe_walk->target_leaf_level) + return xe_pt_hugepte_possible(addr, next, level, xe_walk); + + if (level == xe_walk->target_leaf_level) + return xe_pt_hugepte_possible(addr, next, level, xe_walk); + + return false; +} + +static bool xe_pt_exact_leaf_required_but_invalid(u64 addr, u64 next, + unsigned int level, + struct xe_pt_stage_bind_walk *xe_walk) +{ + struct xe_device *xe = xe_walk->vm->xe; + + if (!xe_debug_page_size_mode_not_none(xe)) + return false; + + return !xe_walk->clear_pt && + xe_walk->target_leaf_level && + level == xe_walk->target_leaf_level && + !xe_pt_hugepte_possible(addr, next, level, xe_walk); +} + static int xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, unsigned int level, u64 addr, u64 next, @@ -531,8 +572,18 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, int ret = 0; u64 pte; - /* Is this a leaf entry ?*/ - if (level == 0 || xe_pt_hugepte_possible(addr, next, level, xe_walk)) { + if (xe_pt_exact_leaf_required_but_invalid(addr, next, level, xe_walk)) + return -EINVAL; + + /* + * Is this a leaf entry? + * Always create a 4K leaf at level 0. For huge pages (level > 0), + * validate alignment and size with xe_pt_hugepte_possible(). + * When target_leaf_level is non-zero, only that huge-page level is + * accepted for normal bind walks. Clear walks remain unconstrained so + * existing huge leaves can be cleared without descending further. + */ + if (level == 0 || xe_pt_huge_leaf_allowed(addr, next, level, xe_walk)) { struct xe_res_cursor *curs = xe_walk->curs; struct xe_bo *bo = xe_vma_bo(xe_walk->vma); bool is_null_or_purged = xe_vma_is_null(xe_walk->vma) || @@ -682,6 +733,26 @@ static bool xe_atomic_for_system(struct xe_vm *vm, struct xe_vma *vma) (bo && xe_bo_has_single_placement(bo)))); } +static u32 xe_pt_target_leaf_level_from_bo(struct xe_device *xe, + struct xe_vma *vma) +{ + struct xe_bo *bo = xe_vma_bo(vma); + + if (!xe_debug_page_size_mode_not_none(xe)) + return 0; + + if (!bo || !xe_bo_is_vram(bo) || !(bo->flags & XE_BO_FLAG_USER)) + return 0; + + if (bo->flags & XE_BO_FLAG_NEEDS_1G) + return 2; + + if (bo->flags & XE_BO_FLAG_NEEDS_2M) + return 1; + + return 0; +} + /** * xe_pt_stage_bind() - Build a disconnected page-table tree for a given address * range. @@ -774,9 +845,13 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, xe_svm_notifier_unlock(vm); } + xe_walk.target_leaf_level = xe_pt_target_leaf_level_from_bo(xe, vma); xe_walk.needs_64K = (vm->flags & XE_VM_FLAG_64K); - if (clear_pt) + if (clear_pt) { + xe_assert(xe, !range); + curs.size = xe_vma_size(vma); goto walk_pt; + } if (vma->gpuva.flags & XE_VMA_ATOMIC_PTE_BIT) { xe_walk.default_vram_pte = xe_atomic_for_vram(vm, vma) ? XE_USM_PPGTT_PTE_AE : 0; @@ -1418,6 +1493,7 @@ static int xe_pt_pre_commit(struct xe_migrate_pt_update *pt_update) pt_update_ops, rftree); } +#if IS_ENABLED(CONFIG_DRM_GPUSVM) /* * Acquire/release the svm notifier_lock around xe_pt_svm_userptr_pre_commit() * and the matching late release in xe_pt_update_ops_run(). Read mode by @@ -1444,6 +1520,10 @@ static void xe_pt_svm_userptr_notifier_unlock(struct xe_vm *vm) xe_svm_notifier_unlock(vm); #endif } +#else +static inline void xe_pt_svm_userptr_notifier_lock(struct xe_vm *vm) { } +static inline void xe_pt_svm_userptr_notifier_unlock(struct xe_vm *vm) { } +#endif #if IS_ENABLED(CONFIG_DRM_GPUSVM) #ifdef CONFIG_DRM_XE_USERPTR_INVAL_INJECT @@ -2366,8 +2446,11 @@ static void xe_pt_update_ops_init(struct xe_vm_pgtable_update_ops *pt_update_ops) { init_llist_head(&pt_update_ops->deferred); + pt_update_ops->current_op = 0; pt_update_ops->start = ~0x0ull; pt_update_ops->last = 0x0ull; + pt_update_ops->needs_svm_lock = false; + pt_update_ops->needs_invalidation = false; xe_page_reclaim_list_init(&pt_update_ops->prl); } diff --git a/drivers/gpu/drm/xe/xe_ras.c b/drivers/gpu/drm/xe/xe_ras.c index 74d5016d9ffe..a31e06b8aa67 100644 --- a/drivers/gpu/drm/xe/xe_ras.c +++ b/drivers/gpu/drm/xe/xe_ras.c @@ -8,12 +8,22 @@ #include "xe_pm.h" #include "xe_printk.h" #include "xe_ras.h" -#include "xe_ras_types.h" +#include "xe_survivability_mode.h" #include "xe_sysctrl.h" #include "xe_sysctrl_event_types.h" #include "xe_sysctrl_mailbox.h" #include "xe_sysctrl_mailbox_types.h" +#define CORE_COMPUTE_UNCORR_TYPE GENMASK(26, 25) +/* + * Uncorrectable error type for core compute errors. + * 0 - Correctable Error + * 1 - Local Uncorrectable Error + * 2 - Global Uncorrectable Error + * 3 - Informational Error + */ +#define GLOBAL_UNCORR_ERROR 2 + /* Severity of detected errors */ enum xe_ras_severity { XE_RAS_SEV_NOT_SUPPORTED = 0, @@ -46,6 +56,14 @@ enum xe_ras_response_status { XE_RAS_STATUS_MAX }; +/* GPU health values */ +enum xe_ras_health { + XE_RAS_HEALTH_OK = 0, + XE_RAS_HEALTH_WARNING, + XE_RAS_HEALTH_CRITICAL, + XE_RAS_HEALTH_MAX +}; + static const char *const xe_ras_severities[] = { [XE_RAS_SEV_NOT_SUPPORTED] = "Not Supported", [XE_RAS_SEV_CORRECTABLE] = "Correctable Error", @@ -65,6 +83,13 @@ static const char *const xe_ras_components[] = { }; static_assert(ARRAY_SIZE(xe_ras_components) == XE_RAS_COMP_MAX); +static const char * const gpu_health_states[] = { + [XE_RAS_HEALTH_OK] = "ok", + [XE_RAS_HEALTH_WARNING] = "warning", + [XE_RAS_HEALTH_CRITICAL] = "critical", +}; +static_assert(ARRAY_SIZE(gpu_health_states) == XE_RAS_HEALTH_MAX); + static u8 drm_to_xe_ras_severity(u8 severity) { switch (severity) { @@ -193,6 +218,94 @@ static void ras_usp_aer_init(struct xe_device *xe) dev_dbg(&usp->dev, "Uncorrectable Internal Errors downgraded and unmasked\n"); } +static u8 handle_core_compute_errors(struct xe_ras_error_array *arr) +{ + struct xe_ras_compute_error *error_info = (void *)arr->details; + u8 uncorr_type; + + uncorr_type = FIELD_GET(CORE_COMPUTE_UNCORR_TYPE, error_info->log_header); + + /* Request a reset if error is global */ + if (uncorr_type == GLOBAL_UNCORR_ERROR) + return XE_RAS_RECOVERY_ACTION_RESET; + + /* + * No action needed for other errors. + * Local errors are recovered using an engine reset by GuC. + */ + return XE_RAS_RECOVERY_ACTION_RECOVERED; +} + +static u8 handle_soc_internal_errors(struct xe_device *xe, struct xe_ras_error_array *arr) +{ + struct xe_ras_soc_error *info = (void *)arr->details; + struct xe_ras_soc_error_source *source = &info->source; + struct xe_ras_error_class *counter = &arr->counter; + + if (source->csc) { + struct xe_ras_csc_error *csc_error = (void *)info->details; + + /* + * CSC uncorrectable errors are classified as hardware errors and firmware errors. + * CSC firmware errors are critical errors that can be recovered only by firmware + * update via SPI driver. On a CSC firmware error, PCODE enables FDO mode and sets + * the bit in the capability register. On receiving this error, the driver enables + * runtime survivability mode which notifies userspace that a firmware update + * is required. + */ + if (csc_error->hec_fw_error) { + xe_err(xe, "[RAS]: CSC %s detected: 0x%x\n", + sev_to_str(counter->common.severity), + csc_error->hec_fw_error); + xe_survivability_mode_runtime_enable(xe); + return XE_RAS_RECOVERY_ACTION_DISCONNECT; + } + } else if (source->ieh) { + struct xe_ras_ieh_error *ieh_error = (void *)info->details; + + if (ieh_error->global_error_status & XE_RAS_SOC_IEH_PUNIT) { + xe_err(xe, "[RAS]: PUNIT %s detected: 0x%x\n", + sev_to_str(counter->common.severity), + ieh_error->global_error_status); + /* TODO: Add PUNIT error handling */ + return XE_RAS_RECOVERY_ACTION_DISCONNECT; + } + } + + /* For other SoC internal errors, request a reset as recovery mechanism */ + return XE_RAS_RECOVERY_ACTION_RESET; +} + +static u8 handle_device_memory_errors(struct xe_device *xe, struct xe_ras_error_array *arr) +{ + struct xe_ras_memory_error *info = (void *)arr->details; + + /* + * For memory errors, the recovery action depends on the error category + * + * TODO: Double-bit ECC errors: Page offlining + * Poison and data parity errors: Log only + * For any other memory errors, request a reset as recovery mechanism + */ + switch (info->category) { + case XE_RAS_MEMORY_POISON: + xe_info(xe, "[RAS]: Poison error detected\n"); + break; + case XE_RAS_MEMORY_DATA_PARITY: + xe_info(xe, "[RAS]: Data parity error detected\n"); + break; + case XE_RAS_MEMORY_DB_ECC: + xe_info(xe, "[RAS]: Double-bit ECC error detected at sw address 0x%llx\n", + info->sw_address); + /* TODO: Add page offlining for Double-bit ECC error */ + fallthrough; + default: + return XE_RAS_RECOVERY_ACTION_RESET; + } + + return XE_RAS_RECOVERY_ACTION_RECOVERED; +} + void xe_ras_counter_threshold_crossed(struct xe_device *xe, struct xe_sysctrl_event_response *response) { @@ -255,6 +368,99 @@ static int get_counter(struct xe_device *xe, struct xe_ras_error_class *counter, } /** + * xe_ras_process_errors() - Process and contain hardware errors + * @xe: xe device instance + * + * Get error details from system controller and return recovery + * method. + * + * Returns: recovery action to be taken + */ +enum xe_ras_recovery_action xe_ras_process_errors(struct xe_device *xe) +{ + struct xe_sysctrl_mailbox_command command = {0}; + enum xe_ras_recovery_action final_action; + u32 remaining = XE_SYSCTRL_FLOOD_LIMIT; + struct xe_ras_get_soc_error response; + size_t rlen; + int ret; + + if (!xe->info.has_sysctrl) + return XE_RAS_RECOVERY_ACTION_RESET; + + /* Default action */ + final_action = XE_RAS_RECOVERY_ACTION_RECOVERED; + + xe_sysctrl_create_command(&command, XE_SYSCTRL_GROUP_GFSP, XE_SYSCTRL_CMD_GET_SOC_ERROR, + NULL, 0, &response, sizeof(response)); + + do { + memset(&response, 0, sizeof(response)); + + ret = xe_sysctrl_send_command(&xe->sc, &command, &rlen); + if (ret) { + xe_err(xe, "sysctrl: failed to get soc error %d\n", ret); + goto err; + } + + if (rlen != sizeof(response)) { + xe_err(xe, "sysctrl: unexpected get soc error response length %zu (expected %zu)\n", + rlen, sizeof(response)); + goto err; + } + + /* Report if number of errors exceeds the maximum errors supported */ + if (response.num_errors > XE_RAS_NUM_ERROR_ARR) + xe_err(xe, "sysctrl: number of errors received %d out of bound (%d)\n", + response.num_errors, XE_RAS_NUM_ERROR_ARR); + + for (int i = 0; i < response.num_errors && i < XE_RAS_NUM_ERROR_ARR; i++) { + struct xe_ras_error_array *arr = &response.arr[i]; + enum xe_ras_recovery_action action; + u8 component, severity; + + component = arr->counter.common.component; + severity = arr->counter.common.severity; + + xe_info(xe, "[RAS]: %s %s detected\n", comp_to_str(component), + sev_to_str(severity)); + + switch (component) { + case XE_RAS_COMP_CORE_COMPUTE: + action = handle_core_compute_errors(arr); + break; + case XE_RAS_COMP_SOC_INTERNAL: + action = handle_soc_internal_errors(xe, arr); + break; + case XE_RAS_COMP_DEVICE_MEMORY: + action = handle_device_memory_errors(xe, arr); + break; + default: + /* For any other component, reset */ + action = XE_RAS_RECOVERY_ACTION_RESET; + break; + } + + /* Process and log all errors and then trigger highest recovery action */ + if (action > final_action) + final_action = action; + } + + /* Treat flooding as a system controller error */ + if (!--remaining) { + xe_err(xe, "[RAS]: sysctrl: get soc error response flooding\n"); + goto err; + } + + } while (response.additional_errors); + + return final_action; + +err: + return XE_RAS_RECOVERY_ACTION_RESET; +} + +/** * xe_ras_get_counter() - Get error counter value * @xe: Xe device instance * @severity: Error severity to be queried (&enum drm_xe_ras_error_severity) @@ -332,6 +538,139 @@ int xe_ras_clear_counter(struct xe_device *xe, u8 severity, u8 component) return 0; } +static ssize_t gpu_health_show(struct device *dev, struct device_attribute *attr, char *buf) +{ + struct xe_ras_get_health_response response = {0}; + struct xe_sysctrl_mailbox_command command = {0}; + struct xe_ras_get_health_request request = {0}; + struct xe_device *xe = kdev_to_xe_device(dev); + const char *health; + size_t rlen; + int ret; + + xe_sysctrl_create_command(&command, XE_SYSCTRL_GROUP_GFSP, XE_SYSCTRL_CMD_GET_HEALTH, + &request, sizeof(request), &response, sizeof(response)); + guard(xe_pm_runtime)(xe); + ret = xe_sysctrl_send_command(&xe->sc, &command, &rlen); + if (ret) { + xe_err(xe, "sysctrl: failed to get health %d\n", ret); + return ret; + } + + if (rlen != sizeof(response)) { + xe_err(xe, "sysctrl: unexpected get health response length %zu (expected %zu)\n", + rlen, sizeof(response)); + return -EIO; + } + if (response.health >= XE_RAS_HEALTH_MAX) { + xe_err(xe, "sysctrl: invalid health state %u\n", + response.health); + return -EIO; + } + + health = gpu_health_states[response.health]; + + xe_dbg(xe, "[RAS]: get health: %s\n", health); + + return sysfs_emit(buf, "%s\n", health); +} + +static ssize_t gpu_health_store(struct device *dev, struct device_attribute *attr, + const char *buf, size_t count) +{ + struct xe_ras_set_health_response response = {0}; + struct xe_sysctrl_mailbox_command command = {0}; + struct xe_ras_set_health_request request = {0}; + struct xe_device *xe = kdev_to_xe_device(dev); + const char *health; + size_t rlen; + int state; + int ret; + + state = sysfs_match_string(gpu_health_states, buf); + if (state < 0) + return -EINVAL; + + request.health = state; + + xe_sysctrl_create_command(&command, XE_SYSCTRL_GROUP_GFSP, XE_SYSCTRL_CMD_SET_HEALTH, + &request, sizeof(request), &response, sizeof(response)); + guard(xe_pm_runtime)(xe); + ret = xe_sysctrl_send_command(&xe->sc, &command, &rlen); + if (ret) { + xe_err(xe, "sysctrl: failed to set health %d\n", ret); + return ret; + } + + if (rlen != sizeof(response)) { + xe_err(xe, "sysctrl: unexpected set health response length %zu (expected %zu)\n", + rlen, sizeof(response)); + return -EIO; + } + + ret = ras_status_to_errno(response.status); + if (ret) { + xe_err(xe, "sysctrl: set health command failed with status %#x\n", + response.status); + return ret; + } + + if (response.health >= XE_RAS_HEALTH_MAX) { + xe_err(xe, "sysctrl: invalid health state %u\n", + response.health); + return -EIO; + } + + health = gpu_health_states[response.health]; + + xe_dbg(xe, "[RAS]: set health: %s\n", health); + + return count; +} +static DEVICE_ATTR_RW(gpu_health); + +static struct attribute *gpu_health_attrs[] = { + &dev_attr_gpu_health.attr, + NULL +}; + +/** + * DOC: GPU Health Indicator + * + * On Intel Xe platforms that support the gpu health indicator interface, + * the driver exposes this sysfs attribute for in-band access to the gpu + * health state:: + * + * /sys/bus/pci/devices/<device>/gpu_health + * + * Reading the attribute is available to all users and returns a single + * line containing the current gpu health state, whereas writing is + * restricted to administrative users and updates the state to one of the + * valid values. + * + * Management tools and administrators use this interface to query the + * current gpu health state (e.g. for telemetry/monitoring) and to + * update it - for example, to mark the gpu as ``warning`` or ``critical`` + * after diagnostics, or reset it back to ``ok`` once remediated. + * + * The valid values for the gpu health state are: + * + * - ``ok`` + * The gpu is healthy and operating within normal parameters. + * + * - ``warning`` + * The gpu is experiencing minor issues but remains operational. + * + * - ``critical`` + * The gpu is in a critical state and may not be operational. + * + * See Documentation/ABI/testing/sysfs-driver-intel-xe-ras for the ABI + * specification. + */ +static const struct attribute_group gpu_health_group = { + .attrs = gpu_health_attrs, +}; + /** * xe_ras_init - Initialize Xe RAS * @xe: xe device instance @@ -340,6 +679,8 @@ int xe_ras_clear_counter(struct xe_device *xe, u8 severity, u8 component) */ void xe_ras_init(struct xe_device *xe) { + int ret; + if (!xe->info.has_drm_ras) return; @@ -350,4 +691,14 @@ void xe_ras_init(struct xe_device *xe) if (IS_ENABLED(CONFIG_PCIEAER)) ras_usp_aer_init(xe); + + /* + * During probe, process and log any errors detected by firmware while the driver was not + * loaded. Critical errors such as Punit and CSC are reported through Pcode init failure, + * causing the driver to enter survivability mode. + */ + xe_ras_process_errors(xe); + ret = devm_device_add_group(xe->drm.dev, &gpu_health_group); + if (ret) + xe_err(xe, "Failed to create GPU health sysfs, err=%d\n", ret); } diff --git a/drivers/gpu/drm/xe/xe_ras.h b/drivers/gpu/drm/xe/xe_ras.h index ba0b0224df23..618364734043 100644 --- a/drivers/gpu/drm/xe/xe_ras.h +++ b/drivers/gpu/drm/xe/xe_ras.h @@ -7,6 +7,7 @@ #define _XE_RAS_H_ #include <linux/types.h> +#include "xe_ras_types.h" struct xe_device; struct xe_sysctrl_event_response; @@ -16,5 +17,6 @@ void xe_ras_counter_threshold_crossed(struct xe_device *xe, int xe_ras_get_counter(struct xe_device *xe, u8 severity, u8 component, u32 *value); int xe_ras_clear_counter(struct xe_device *xe, u8 severity, u8 component); void xe_ras_init(struct xe_device *xe); +enum xe_ras_recovery_action xe_ras_process_errors(struct xe_device *xe); #endif diff --git a/drivers/gpu/drm/xe/xe_ras_types.h b/drivers/gpu/drm/xe/xe_ras_types.h index 6688e11f57a8..99b2466e2062 100644 --- a/drivers/gpu/drm/xe/xe_ras_types.h +++ b/drivers/gpu/drm/xe/xe_ras_types.h @@ -9,6 +9,31 @@ #include <linux/types.h> #define XE_RAS_NUM_COUNTERS 16 +#define XE_RAS_NUM_ERROR_ARR 3 +/* Error bits in IEH global error status register */ +#define XE_RAS_SOC_IEH_PUNIT BIT(1) +/* Device memory error categories */ +#define XE_RAS_MEMORY_DB_ECC BIT(1) +#define XE_RAS_MEMORY_POISON BIT(2) +#define XE_RAS_MEMORY_DATA_PARITY BIT(5) + +/** + * enum xe_ras_recovery_action - RAS recovery actions + * + * @XE_RAS_RECOVERY_ACTION_RECOVERED: Error recovered + * @XE_RAS_RECOVERY_ACTION_RESET: Requires reset + * @XE_RAS_RECOVERY_ACTION_DISCONNECT: Requires disconnect + * @XE_RAS_RECOVERY_ACTION_MAX: Max action value + * + * This enum defines the possible recovery actions that can be taken in response + * to RAS errors. + */ +enum xe_ras_recovery_action { + XE_RAS_RECOVERY_ACTION_RECOVERED = 0, + XE_RAS_RECOVERY_ACTION_RESET, + XE_RAS_RECOVERY_ACTION_DISCONNECT, + XE_RAS_RECOVERY_ACTION_MAX +}; /** * struct xe_ras_error_common - Error fields that are common across all products @@ -121,4 +146,144 @@ struct xe_ras_clear_counter_response { /** @reserved1: Reserved for future use */ u32 reserved1[3]; } __packed; + +/** + * struct xe_ras_error_array - Details of the error types + */ +struct xe_ras_error_array { + /** @value: Counter value of the detailed error */ + u32 value; + /** @counter: Error counter */ + struct xe_ras_error_class counter; + /** @timestamp: Timestamp */ + u64 timestamp; + /** @details: Error details specific to the counter */ + u32 details[XE_RAS_NUM_COUNTERS]; +} __packed; + +/** + * struct xe_ras_get_soc_error - Response from get soc error command + */ +struct xe_ras_get_soc_error { + /** @num_errors: Number of errors reported in this response */ + u8 num_errors; + /** @additional_errors: Indicates if the errors are pending */ + u8 additional_errors; + /** @arr: Array of up to 3 errors */ + struct xe_ras_error_array arr[XE_RAS_NUM_ERROR_ARR]; +} __packed; + +/** + * struct xe_ras_compute_error - Error details of Core Compute error + */ +struct xe_ras_compute_error { + /** @log_header: Error Source and type */ + u32 log_header; + /** @reserved: Reserved */ + u32 reserved[15]; +} __packed; + +/** + * struct xe_ras_soc_error_source - Source of SoC error + */ +struct xe_ras_soc_error_source { + /** @csc: CSC */ + u32 csc:1; + /** @ieh: IEH (Integrated Error Handler) */ + u32 ieh:1; + /** @reserved: Reserved for future use */ + u32 reserved:30; +} __packed; + +/** + * struct xe_ras_soc_error - Error details of SoC internal error + */ +struct xe_ras_soc_error { + /** @source: Error source */ + struct xe_ras_soc_error_source source; + /** @details: Error details specific to the error source */ + u32 details[15]; +} __packed; + +/** + * struct xe_ras_csc_error - CSC error details + */ +struct xe_ras_csc_error { + /** @reserved: Reserved for future use */ + u32 reserved; + /** @hec_fw_error: CSC firmware error */ + u32 hec_fw_error; +} __packed; + +/** + * struct xe_ras_ieh_error - IEH (Integrated Error Handler) error details + */ +struct xe_ras_ieh_error { + /** @reserved: Reserved for future use */ + u32 reserved; + /** @global_error_status: Global error status */ + u32 global_error_status; + /** @reserved1: Reserved for future use */ + u32 reserved1[2]; + /** @info: Additional information */ + u32 info[10]; +} __packed; + +/** + * struct xe_ras_memory_error - Device memory error details + */ +struct xe_ras_memory_error { + /** @category: Device memory error category */ + u8 category; + /** @reserved: Reserved for future use */ + u8 reserved[7]; + /** @reserved1: Reserved for future use */ + u64 reserved1; + /** @sw_address: Software address where error occurred */ + u64 sw_address; + /** @reserved2: Reserved for future use */ + u32 reserved2[10]; +} __packed; + +/** + * struct xe_ras_get_health_request - Request structure for obtaining gpu health + */ +struct xe_ras_get_health_request { + /** @reserved: Reserved for future use. */ + u32 reserved[2]; +} __packed; + +/** + * struct xe_ras_get_health_response - Response structure for obtaining gpu health + */ +struct xe_ras_get_health_response { + /** @health: gpu health value */ + u8 health; + /** @reserved: Reserved for future use */ + u8 reserved[3]; +} __packed; + +/** + * struct xe_ras_set_health_request - Request structure for setting gpu health + */ +struct xe_ras_set_health_request { + /** @health: gpu health value */ + u8 health; + /** @reserved: Reserved for future use */ + u8 reserved[3]; +} __packed; + +/** + * struct xe_ras_set_health_response - Response structure for setting gpu health + */ +struct xe_ras_set_health_response { + /** @status: Status of set health operation */ + u32 status; + /** @health: Resulting gpu health value */ + u8 health; + /** @reserved: Reserved for future use */ + u8 reserved[3]; + /** @reserved1: Reserved for future use */ + u32 reserved1[2]; +} __packed; #endif diff --git a/drivers/gpu/drm/xe/xe_sriov_packet.c b/drivers/gpu/drm/xe/xe_sriov_packet.c index 2ae9eff2a7c0..e9ae9c9744ea 100644 --- a/drivers/gpu/drm/xe/xe_sriov_packet.c +++ b/drivers/gpu/drm/xe/xe_sriov_packet.c @@ -3,6 +3,8 @@ * Copyright © 2025 Intel Corporation */ +#include "abi/xe_driver_klvs_abi.h" + #include "xe_bo.h" #include "xe_device.h" #include "xe_guc_klv_helpers.h" @@ -352,19 +354,13 @@ ssize_t xe_sriov_packet_write_single(struct xe_device *xe, unsigned int vfid, return copied; } -#define MIGRATION_KLV_DEVICE_DEVID_KEY 0xf001u -#define MIGRATION_KLV_DEVICE_DEVID_LEN 1u -#define MIGRATION_KLV_DEVICE_REVID_KEY 0xf002u -#define MIGRATION_KLV_DEVICE_REVID_LEN 1u - #define MIGRATION_DESCRIPTOR_DWORDS (GUC_KLV_LEN_MIN + MIGRATION_KLV_DEVICE_DEVID_LEN + \ GUC_KLV_LEN_MIN + MIGRATION_KLV_DEVICE_REVID_LEN) static int pf_descriptor_init(struct xe_device *xe, unsigned int vfid) { struct xe_sriov_packet **desc = pf_pick_descriptor(xe, vfid); struct xe_sriov_packet *data; - unsigned int len = 0; - u32 *klvs; + u32 *klvs, *end; int ret; data = xe_sriov_packet_alloc(xe); @@ -379,20 +375,55 @@ static int pf_descriptor_init(struct xe_device *xe, unsigned int vfid) } klvs = data->vaddr; - klvs[len++] = PREP_GUC_KLV_CONST(MIGRATION_KLV_DEVICE_DEVID_KEY, - MIGRATION_KLV_DEVICE_DEVID_LEN); - klvs[len++] = xe->info.devid; - klvs[len++] = PREP_GUC_KLV_CONST(MIGRATION_KLV_DEVICE_REVID_KEY, - MIGRATION_KLV_DEVICE_REVID_LEN); - klvs[len++] = xe->info.revid; + end = klvs + MIGRATION_DESCRIPTOR_DWORDS; - xe_assert(xe, len == MIGRATION_DESCRIPTOR_DWORDS); + klvs = xe_guc_klv_encode_u32(klvs, end - klvs, + MIGRATION_KLV_DEVICE_DEVID_KEY, + xe->info.devid); + klvs = xe_guc_klv_encode_u32(klvs, end - klvs, + MIGRATION_KLV_DEVICE_REVID_KEY, + xe->info.revid); + xe_assert(xe, !IS_ERR(klvs)); + xe_assert(xe, klvs == end); *desc = data; return 0; } +static int descriptor_decoder(void *arg, u16 key, u16 len, const u32 *value) +{ + struct xe_device *xe = arg; + + xe_sriov_dbg_verbose(xe, "found KLV %#x %s\n", key, xe_guc_klv_key_to_string(key)); + + switch (key) { + case MIGRATION_KLV_DEVICE_DEVID_KEY: + if (*value != xe->info.devid) { + xe_sriov_warn(xe, "Aborting migration, devid mismatch %#06x!=%#06x\n", + *value, xe->info.devid); + return -ENODEV; + } + break; + case MIGRATION_KLV_DEVICE_REVID_KEY: + if (*value != xe->info.revid) { + xe_sriov_warn(xe, "Aborting migration, revid mismatch %#06x!=%#06x\n", + *value, xe->info.revid); + return -ENODEV; + } + break; + default: + if (IS_ENABLED(CONFIG_DRM_XE_DEBUG)) { + struct drm_printer p = xe_dbg_printer(xe); + + xe_sriov_dbg(xe, "unexpected KLV %#x in descriptor!\n", key); + xe_guc_klv_print_one(key, len, value, &p); + } + return 0; + } + return 1; +} + /** * xe_sriov_packet_process_descriptor() - Process migration data descriptor packet. * @xe: the &xe_device @@ -409,6 +440,7 @@ int xe_sriov_packet_process_descriptor(struct xe_device *xe, unsigned int vfid, { u32 num_dwords = data->hdr.size / sizeof(u32); u32 *klvs = data->vaddr; + int ret; xe_assert(xe, data->hdr.type == XE_SRIOV_PACKET_TYPE_DESCRIPTOR); @@ -418,47 +450,18 @@ int xe_sriov_packet_process_descriptor(struct xe_device *xe, unsigned int vfid, return -EINVAL; } - while (num_dwords >= GUC_KLV_LEN_MIN) { - u32 key = FIELD_GET(GUC_KLV_0_KEY, klvs[0]); - u32 len = FIELD_GET(GUC_KLV_0_LEN, klvs[0]); - - klvs += GUC_KLV_LEN_MIN; - num_dwords -= GUC_KLV_LEN_MIN; - - if (len > num_dwords) { - xe_sriov_warn(xe, "Aborting migration, truncated KLV %#x, len %u\n", - key, len); - return -EINVAL; - } - - switch (key) { - case MIGRATION_KLV_DEVICE_DEVID_KEY: - if (*klvs != xe->info.devid) { - xe_sriov_warn(xe, - "Aborting migration, devid mismatch %#06x!=%#06x\n", - *klvs, xe->info.devid); - return -ENODEV; - } - break; - case MIGRATION_KLV_DEVICE_REVID_KEY: - if (*klvs != xe->info.revid) { - xe_sriov_warn(xe, - "Aborting migration, revid mismatch %#06x!=%#06x\n", - *klvs, xe->info.revid); - return -ENODEV; - } - break; - default: - xe_sriov_dbg(xe, - "Skipping unknown migration KLV %#x, len=%u\n", - key, len); - print_hex_dump_bytes("desc: ", DUMP_PREFIX_OFFSET, klvs, - min(SZ_64, len * sizeof(u32))); - break; - } + ret = xe_guc_klv_count(klvs, num_dwords); + if (ret < 0) { + xe_sriov_warn(xe, "Aborting migration, corrupted descriptor KLVs (%pe)\n", + ERR_PTR(ret)); + return ret; + } - klvs += len; - num_dwords -= len; + ret = xe_guc_klv_parser(klvs, num_dwords, xe, descriptor_decoder); + if (ret < 0) { + xe_sriov_warn(xe, "Aborting migration, descriptor parsing failed (%pe)\n", + ERR_PTR(ret)); + return ret; } return 0; @@ -519,3 +522,7 @@ int xe_sriov_packet_save_init(struct xe_device *xe, unsigned int vfid) return 0; } + +#if IS_BUILTIN(CONFIG_DRM_XE_KUNIT_TEST) +#include "tests/xe_sriov_packet_kunit.c" +#endif diff --git a/drivers/gpu/drm/xe/xe_sriov_vf_ccs.c b/drivers/gpu/drm/xe/xe_sriov_vf_ccs.c index 09b99fb2608b..a8c831fbee3b 100644 --- a/drivers/gpu/drm/xe/xe_sriov_vf_ccs.c +++ b/drivers/gpu/drm/xe/xe_sriov_vf_ccs.c @@ -3,6 +3,8 @@ * Copyright © 2025 Intel Corporation */ +#include <drm/drm_drv.h> + #include "instructions/xe_mi_commands.h" #include "instructions/xe_gpu_commands.h" #include "xe_bb.h" @@ -404,6 +406,8 @@ void xe_sriov_vf_ccs_rw_update_bb_addr(struct xe_sriov_vf_ccs_ctx *ctx) /** * xe_sriov_vf_ccs_attach_bo - Insert CCS read write commands in the BO. * @bo: the &buffer object to which batch buffer commands will be added. + * @new_mem: the (not yet committed) destination resource @bo is being moved + * into; bo->ttm.resource is still the old resource at this point. * * This function shall be called only by VF. It inserts the PTEs and copy * command instructions in the BO by calling xe_migrate_ccs_rw_copy() @@ -411,7 +415,7 @@ void xe_sriov_vf_ccs_rw_update_bb_addr(struct xe_sriov_vf_ccs_ctx *ctx) * * Returns: 0 if successful, negative error code on failure. */ -int xe_sriov_vf_ccs_attach_bo(struct xe_bo *bo) +int xe_sriov_vf_ccs_attach_bo(struct xe_bo *bo, struct ttm_resource *new_mem) { struct xe_device *xe = xe_bo_device(bo); enum xe_sriov_vf_ccs_rw_ctxs ctx_id; @@ -430,7 +434,21 @@ int xe_sriov_vf_ccs_attach_bo(struct xe_bo *bo) xe_assert(xe, !bb); ctx = &xe->sriov.vf.ccs.contexts[ctx_id]; - err = xe_migrate_ccs_rw_copy(tile, ctx->mig_q, bo, ctx_id); + err = xe_migrate_ccs_rw_copy(tile, ctx->mig_q, bo, new_mem, ctx_id); + if (err) + goto err_unwind; + } + return 0; + +err_unwind: + /* + * Clean up any contexts already attached. Can't reuse + * xe_sriov_vf_ccs_detach_bo() here as it requires both contexts + * attached before cleaning up either one. + */ + for_each_ccs_rw_ctx(ctx_id) { + if (bo->bb_ccs[ctx_id]) + xe_migrate_ccs_rw_copy_clear(bo, ctx_id, true); } return err; } @@ -450,19 +468,27 @@ int xe_sriov_vf_ccs_detach_bo(struct xe_bo *bo) struct xe_device *xe = xe_bo_device(bo); enum xe_sriov_vf_ccs_rw_ctxs ctx_id; struct xe_mem_pool_node *bb; + bool bound; + int idx; xe_assert(xe, IS_VF_CCS_READY(xe)); if (!xe_bo_has_valid_ccs_bb(bo)) return 0; + bound = drm_dev_enter(&xe->drm, &idx); + for_each_ccs_rw_ctx(ctx_id) { bb = bo->bb_ccs[ctx_id]; if (!bb) continue; - xe_migrate_ccs_rw_copy_clear(bo, ctx_id); + xe_migrate_ccs_rw_copy_clear(bo, ctx_id, bound); } + + if (bound) + drm_dev_exit(idx); + return 0; } diff --git a/drivers/gpu/drm/xe/xe_sriov_vf_ccs.h b/drivers/gpu/drm/xe/xe_sriov_vf_ccs.h index 00e58b36c510..e1034d852104 100644 --- a/drivers/gpu/drm/xe/xe_sriov_vf_ccs.h +++ b/drivers/gpu/drm/xe/xe_sriov_vf_ccs.h @@ -11,11 +11,12 @@ #include "xe_sriov_vf_ccs_types.h" struct drm_printer; +struct ttm_resource; struct xe_device; struct xe_bo; int xe_sriov_vf_ccs_init(struct xe_device *xe); -int xe_sriov_vf_ccs_attach_bo(struct xe_bo *bo); +int xe_sriov_vf_ccs_attach_bo(struct xe_bo *bo, struct ttm_resource *new_mem); int xe_sriov_vf_ccs_detach_bo(struct xe_bo *bo); int xe_sriov_vf_ccs_register_context(struct xe_device *xe); void xe_sriov_vf_ccs_rebase(struct xe_device *xe); diff --git a/drivers/gpu/drm/xe/xe_survivability_mode.c b/drivers/gpu/drm/xe/xe_survivability_mode.c index 427afd144f3a..4c506027fa94 100644 --- a/drivers/gpu/drm/xe/xe_survivability_mode.c +++ b/drivers/gpu/drm/xe/xe_survivability_mode.c @@ -54,7 +54,6 @@ * # cat /sys/bus/pci/devices/<device>/survivability_mode * Boot * - * * Any additional debug information if present will be visible under the directory * ``survivability_info``:: * @@ -98,6 +97,15 @@ * # cat /sys/bus/pci/devices/<device>/survivability_mode * Runtime * + * On some CSC firmware errors, PCODE sets FDO mode and the only recovery possible is through + * firmware flash using SPI driver. Userspace can check if FDO mode is set by checking the below + * sysfs entry. + * + * .. code-block:: shell + * + * # cat /sys/bus/pci/devices/<device>/survivability_info/fdo_mode + * enabled + * * When such errors occur, userspace is notified with the drm device wedged uevent and runtime * survivability mode. User can then initiate a firmware flash using userspace tools like fwupd * to restore device to normal operation. @@ -296,7 +304,8 @@ static int create_survivability_sysfs(struct pci_dev *pdev) if (ret) return ret; - if (check_boot_failure(xe)) { + /* Survivability info is not required if enabled via configfs */ + if (!xe_configfs_get_survivability_mode(pdev)) { ret = devm_device_add_group(dev, &survivability_info_group); if (ret) return ret; diff --git a/drivers/gpu/drm/xe/xe_sysctrl_event.c b/drivers/gpu/drm/xe/xe_sysctrl_event.c index b4d17329af6c..da395148ee9d 100644 --- a/drivers/gpu/drm/xe/xe_sysctrl_event.c +++ b/drivers/gpu/drm/xe/xe_sysctrl_event.c @@ -16,7 +16,7 @@ static void get_pending_event(struct xe_sysctrl *sc, struct xe_sysctrl_mailbox_c { struct xe_sysctrl_event_response *response = command->data_out; struct xe_device *xe = sc_to_xe(sc); - u32 count = XE_SYSCTRL_EVENT_FLOOD; + u32 count = XE_SYSCTRL_FLOOD_LIMIT; size_t len; int ret; diff --git a/drivers/gpu/drm/xe/xe_sysctrl_event_types.h b/drivers/gpu/drm/xe/xe_sysctrl_event_types.h index c16c66b9fa7f..348768ca454a 100644 --- a/drivers/gpu/drm/xe/xe_sysctrl_event_types.h +++ b/drivers/gpu/drm/xe/xe_sysctrl_event_types.h @@ -10,9 +10,6 @@ #define XE_SYSCTRL_EVENT_DATA_LEN 59 -/* Modify as needed */ -#define XE_SYSCTRL_EVENT_FLOOD 16 - /** * enum xe_sysctrl_event - Events reported by System Controller * diff --git a/drivers/gpu/drm/xe/xe_sysctrl_mailbox_types.h b/drivers/gpu/drm/xe/xe_sysctrl_mailbox_types.h index 6e3753554510..d0341538ad05 100644 --- a/drivers/gpu/drm/xe/xe_sysctrl_mailbox_types.h +++ b/drivers/gpu/drm/xe/xe_sysctrl_mailbox_types.h @@ -22,14 +22,20 @@ enum xe_sysctrl_group { /** * enum xe_sysctrl_gfsp_cmd - Commands supported by GFSP group * + * @XE_SYSCTRL_CMD_GET_SOC_ERROR: Retrieve basic error information * @XE_SYSCTRL_CMD_GET_COUNTER: Get error counter value * @XE_SYSCTRL_CMD_CLEAR_COUNTER: Clear error counter value * @XE_SYSCTRL_CMD_GET_PENDING_EVENT: Retrieve pending event + * @XE_SYSCTRL_CMD_GET_HEALTH: Retrieve gpu health + * @XE_SYSCTRL_CMD_SET_HEALTH: Set gpu health */ enum xe_sysctrl_gfsp_cmd { + XE_SYSCTRL_CMD_GET_SOC_ERROR = 0x01, XE_SYSCTRL_CMD_GET_COUNTER = 0x03, XE_SYSCTRL_CMD_CLEAR_COUNTER = 0x04, XE_SYSCTRL_CMD_GET_PENDING_EVENT = 0x07, + XE_SYSCTRL_CMD_GET_HEALTH = 0x0B, + XE_SYSCTRL_CMD_SET_HEALTH = 0x0C, }; /** @@ -52,6 +58,9 @@ struct xe_sysctrl_mailbox_command { size_t data_out_len; }; +/* Modify as needed */ +#define XE_SYSCTRL_FLOOD_LIMIT 16 + #define XE_SYSCTRL_MB_FRAME_SIZE 16 #define XE_SYSCTRL_MB_MAX_FRAMES 64 #define XE_SYSCTRL_MB_MAX_MESSAGE_SIZE \ diff --git a/drivers/gpu/drm/xe/xe_trace.h b/drivers/gpu/drm/xe/xe_trace.h index 750fa32c13b2..2fe8f89a1e34 100644 --- a/drivers/gpu/drm/xe/xe_trace.h +++ b/drivers/gpu/drm/xe/xe_trace.h @@ -213,6 +213,11 @@ DEFINE_EVENT(xe_exec_queue, xe_exec_queue_memory_cat_error, TP_ARGS(q) ); +DEFINE_EVENT(xe_exec_queue, xe_guc_uncorrectable_error, + TP_PROTO(struct xe_exec_queue *q), + TP_ARGS(q) +); + DEFINE_EVENT(xe_exec_queue, xe_exec_queue_cgp_context_error, TP_PROTO(struct xe_exec_queue *q), TP_ARGS(q) diff --git a/drivers/gpu/drm/xe/xe_uc_fw.c b/drivers/gpu/drm/xe/xe_uc_fw.c index 3f08a3b54062..a8e6f18cc9b4 100644 --- a/drivers/gpu/drm/xe/xe_uc_fw.c +++ b/drivers/gpu/drm/xe/xe_uc_fw.c @@ -115,6 +115,7 @@ struct fw_blobs_by_type { #define XE_GT_TYPE_ANY XE_GT_TYPE_UNINITIALIZED #define XE_GUC_FIRMWARE_DEFS(fw_def, mmp_ver, major_ver) \ + fw_def(NOVALAKE_S, GT_TYPE_ANY, major_ver(xe, guc, nvl, 70, 71, 0)) \ fw_def(PANTHERLAKE, GT_TYPE_ANY, major_ver(xe, guc, ptl, 70, 54, 0)) \ fw_def(BATTLEMAGE, GT_TYPE_ANY, major_ver(xe, guc, bmg, 70, 54, 0)) \ fw_def(LUNARLAKE, GT_TYPE_ANY, major_ver(xe, guc, lnl, 70, 53, 0)) \ diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 73ac031ffb04..9e0176861cb6 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -206,7 +206,23 @@ static void resume_and_reinstall_preempt_fences(struct xe_vm *vm, xe_vm_assert_held(vm); list_for_each_entry(q, &vm->preempt.exec_queues, lr.link) { - q->ops->resume(q); + /* + * Only resume queues whose suspend() actually succeeded. A + * failed suspend() (e.g. killed/banned/wedged) leaves the queue + * un-suspended, so it must not be resumed. + * + * Also skip queues that have since been reset/killed/banned/ + * wedged: their suspend may not have completed (suspend_pending + * can still be set, e.g. a preempt fence signalled with -ENOENT + * without waiting), so resuming would trip the !suspend_pending + * assert in the backend. Such queues are being torn down anyway, + * so leave them marked suspended and let teardown resolve their + * state. + */ + if (READ_ONCE(q->lr.suspended) && !q->ops->reset_status(q)) { + WRITE_ONCE(q->lr.suspended, false); + q->ops->resume(q); + } drm_gpuvm_resv_add_fence(&vm->gpuvm, exec, q->lr.pfence, DMA_RESV_USAGE_BOOKKEEP, DMA_RESV_USAGE_BOOKKEEP); @@ -1629,7 +1645,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef) if (xef) vm->xef = xe_file_get(xef); - /** + /* * GSC VMs are kernel-owned, only used for PXP ops and can sometimes be * manipulated under the PXP mutex. However, the PXP mutex can be taken * under a user-VM lock when the PXP session is started at exec_queue @@ -1749,10 +1765,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef) vm->batch_invalidate_tlb = true; } - if (vm->flags & XE_VM_FLAG_LR_MODE) { - INIT_WORK(&vm->preempt.rebind_work, preempt_rebind_work_func); + if (vm->flags & XE_VM_FLAG_LR_MODE) vm->batch_invalidate_tlb = false; - } /* Fill pt_root after allocating scratch tables */ for_each_tile(tile, xe, id) { @@ -1809,10 +1823,10 @@ err_close: return ERR_PTR(err); err_svm_fini: - if (flags & XE_VM_FLAG_FAULT_MODE) { - vm->size = 0; /* close the vm */ - xe_svm_fini(vm); - } + vm->size = 0; /* close the vm */ + if (flags & XE_VM_FLAG_FAULT_MODE) + xe_svm_close(vm); + xe_svm_fini(vm); err_no_resv: mutex_destroy(&vm->snap_mutex); for_each_tile(tile, xe, id) @@ -3255,11 +3269,26 @@ static int op_lock_and_prep(struct drm_exec *exec, struct xe_vm *vm, .request_decompress = false, .check_purged = true, }); - if (!err && !xe_vma_has_no_bo(vma)) - err = xe_bo_migrate(xe_vma_bo(vma), - region_to_mem_type[region], - NULL, - exec); + if (!err && !xe_vma_has_no_bo(vma)) { + struct xe_bo *bo = xe_vma_bo(vma); + u32 mem_type; + + if (region == DRM_XE_CONSULT_MEM_ADVISE_PREF_LOC) { + unsigned int i; + + mem_type = XE_PL_TT; + for (i = 0; i < bo->placement.num_placement; i++) { + if (mem_type_is_vram(bo->placements[i].mem_type)) { + mem_type = bo->placements[i].mem_type; + break; + } + } + } else { + mem_type = region_to_mem_type[region]; + } + + err = xe_bo_migrate(bo, mem_type, NULL, exec); + } break; } default: diff --git a/drivers/gpu/drm/xe/xe_vm_madvise.c b/drivers/gpu/drm/xe/xe_vm_madvise.c index 9e343f9aa44d..0474768a38aa 100644 --- a/drivers/gpu/drm/xe/xe_vm_madvise.c +++ b/drivers/gpu/drm/xe/xe_vm_madvise.c @@ -657,7 +657,7 @@ int xe_vm_madvise_ioctl(struct drm_device *dev, void *data, struct drm_file *fil xe_device_is_l2_flush_optimized(xe) && (pat_index != 19 && coh_mode != XE_COH_2WAY))) { err = -EINVAL; - goto madv_fini; + goto free_vmas; } } diff --git a/drivers/gpu/drm/xe/xe_wa_oob.rules b/drivers/gpu/drm/xe/xe_wa_oob.rules index 9027365f0043..f02ac9bf7424 100644 --- a/drivers/gpu/drm/xe/xe_wa_oob.rules +++ b/drivers/gpu/drm/xe/xe_wa_oob.rules @@ -66,3 +66,9 @@ 14025883347 MEDIA_VERSION_RANGE(1301, 3503) GRAPHICS_VERSION_RANGE(2004, 3005) 16029380221 MEDIA_VERSION(3500) +22022079272 MEDIA_VERSION(3503) + GRAPHICS_VERSION(3510) + GRAPHICS_VERSION(3511) +16029897822 MEDIA_VERSION(3500) + GRAPHICS_VERSION(3510) +14027054324 GRAPHICS_VERSION(3511) diff --git a/drivers/gpu/drm/xe/xe_wopcm.c b/drivers/gpu/drm/xe/xe_wopcm.c index 900daf1d1b1b..fe65ed246775 100644 --- a/drivers/gpu/drm/xe/xe_wopcm.c +++ b/drivers/gpu/drm/xe/xe_wopcm.c @@ -49,9 +49,9 @@ */ /* Default WOPCM size is 2MB from Gen11, 1MB on previous platforms */ -/* FIXME: Larger size require for 2 tile PVC, do a proper probe sooner or later */ +/* FIXME: Larger size require for some platforms, do a proper probe sooner or later */ #define DGFX_WOPCM_SIZE SZ_4M -/* FIXME: Larger size require for MTL, do a proper probe sooner or later */ +#define LNL_WOPCM_SIZE SZ_8M #define MTL_WOPCM_SIZE SZ_4M #define WOPCM_SIZE SZ_2M @@ -179,9 +179,14 @@ err_out: u32 xe_wopcm_size(struct xe_device *xe) { - return IS_DGFX(xe) ? DGFX_WOPCM_SIZE : - xe->info.platform == XE_METEORLAKE ? MTL_WOPCM_SIZE : - WOPCM_SIZE; + if (xe->info.platform >= XE_LUNARLAKE) + return LNL_WOPCM_SIZE; + else if (IS_DGFX(xe)) + return DGFX_WOPCM_SIZE; + else if (xe->info.platform == XE_METEORLAKE) + return MTL_WOPCM_SIZE; + else + return WOPCM_SIZE; } static u32 max_wopcm_size(struct xe_device *xe) diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h index 50c80af4ad4e..509202a7b13e 100644 --- a/include/uapi/drm/xe_drm.h +++ b/include/uapi/drm/xe_drm.h @@ -2537,21 +2537,21 @@ struct drm_xe_exec_queue_set_property { * Refer to Documentation/netlink/specs/drm_ras.yaml for complete interface specification. * * Node Registration - * ================= + * ----------------- * * The driver registers DRM RAS nodes for each error severity level. * enum drm_xe_ras_error_severity defines the node-id, while DRM_XE_RAS_ERROR_SEVERITY_NAMES maps * node-id to node-name. * * Error Classification - * ==================== + * -------------------- * * Each node contains a list of error counters. Each error is identified by a error-id and * an error-name. enum drm_xe_ras_error_component defines the error-id, while * DRM_XE_RAS_ERROR_COMPONENT_NAMES maps error-id to error-name. * * User Interface - * ============== + * -------------- * * To retrieve error values of a error counter, userspace applications should * follow the below steps: |